Stability, complex modes, and nonseparability in rotating quadratic potentials