THE IMACS CLUSTER BUILDING SURVEY. II. SPECTRAL EVOLUTION OF GALAXIES IN THE EPOCH OF CLUSTER ASSEMBLY