Microscopic origin of the Drude-Smith model