A cautionary tale: limitations of a brightness-based spectroscopic approach to chromatic exoplanet radii