The Electromagnetic Duality Formulation of Geometric Phases