Symmetry breaking of fluctuation dynamics by noise color