Constraints on a Universal Stellar Initial Mass Function from Ultraviolet to Near-Infrared Galaxy Luminosity Densities