We have investigated the carrier dynamics in annealed quantum dots where the energy-level separation of the optical transitions can be tuned between 68 and 19 meV. Photoluminescence transients obtained for different excitation densities for the ground and excited states are well described by a random population model, and values of the exciton lifetimes extracted. The ground-state radiative lifetimes are found to vary from 800 ps in the as-grown sample to 490 ps in the sample annealed at the highest temperature. This is attributed to shifts in the emission energy and changes in the electron and hole wave-function overlap as the dot size increases with annealing. We find no evidence of inhibited relaxation of carriers (phonon bottleneck) when the intersublevel energy-level separation is much less than the LO-phonon energies. The rise times of the transients remain fast at low excitation density, where the probability of Coulomb scattering is expected to be low, meaning that another mechanism, possibly multiphonon scattering, is responsible for the fast relaxation observed in self-assembled quantum dots.