Non-Markovian equilibration controlled by symmetry breaking