Foreground mismodeling and the point source explanation of the Fermi Galactic Center excess