Implications of a Temperature-dependent Initial Mass Function. III. Mass Growth and Quiescence