Magnetic-field symmetry breaking in spin glasses