Anomalous reduction of the two‐phonon pinning interaction in magneto‐optics