The Phantom menaced: Constraints on low-energy effective ghosts