The diffusion of stars through phase space