The Rest-Frame Optical Luminosity Density, Color, and Stellar Mass Density of the Universe from z = 0 to z = 3