The Optical and Near-Infrared Colors of Galaxies. II. Spectral Classification