On the evolution of slowly accreting neutron stars