Theory of spin diffusion of dilute, polarized fermions for all temperatures