Augmenting Language Models with Long-Term Memory