Damping of collective vibrations in a memory-dependent transport model