Long-lived dark exciton states are formed in self-assembled quantum dots due to the combination of the angular momentum of electrons and holes. The lifetime of dark excitons are determined by spin-flip processes that transfer dark excitons into radiative bright excitons. We employ time-resolved spontaneous emission measurements in a modified local density of optical states to unambiguously record the spin-flip rate. Pronounced variations in the spin-flip rate with the quantum dot emission energy are observed demonstrating that the exciton storage time can be extended by controlling the quantum dot size. The energy dependence is compared to a recent model from the literature, in which the spin flip is due to the combined action of short-range exchange interaction and acoustic phonons. We furthermore observe a pronounced enhancement of the spin-flip rate close to semiconductor-air interfaces, which illustrates the important role of interfaces for quantum dot based nanophotonic structures.