What the Milky Way’s Dwarfs tell us about the Galactic Center extended excess