Learning Barrier Functions With Memory for Robust Safe Navigation