The Formation Timescales of Giant Spheroids