Microscopic derivation of non-Markovian thermalization of a Brownian particle