Characterizing the spin state of an atomic ensemble using the magneto-optical resonance method