We consider the initialization of the spin state of a single electron trapped in a self-assembled quantum dot via optical pumping of a trion level. We show that with a magnetic field applied perpendicular to the growth direction of the dot, a near-unity fidelity can be obtained in a time equal to a few times the inverse of the spin-conserving trion relaxation rate. This method is several orders of magnitude faster than with the field aligned parallel, since this configuration must rely on a slow hole spin-flip mechanism. This increase in speed does result in a limit on the maximum obtainable fidelity, but we show that for InAs dots, the error is very small.