MINOR MERGERS OR PROGENITOR BIAS? THE STELLAR AGES OF SMALL AND LARGE QUENCHED GALAXIES