What the Milky Way’s dwarfs tell us about the Galactic Center extended gamma-ray excess