Intrinsic characteristic times in the drift-diffusion problem