Adiabatic holonomy phase and vanishing magnetic fields