Memory damping and energy-diffusion-controlled escape