Berry phase in magnetic systems with point perturbations