Gauge invariant determination of charged hadron masses