Including memory banks in a natural language processing architecture increases model capacity by equipping it with additional data at inference time. In this paper, we build upon kNNLM (Khandelwal et al., 2020), which uses a pre-trained language model together with an exhaustive kNN search through the training data (memory bank) to achieve state-of-the-art results. We investigate whether we can improve the kNN-LM performance by instead training a LM with the knowledge that we will be using a kNN post-hoc. We achieved significant improvement using our method on language modeling tasks on WIKI-2 and WIKI-103. The main phenomenon that we encounter is that adding a simple L2 regularization on the activations (not weights) of the model, a transformer, improves the post-hoc kNN classification performance. We explore some possible reasons for this improvement. In particular, we find that the added L2 regularization seems to improve the performance for high-frequency words without deteriorating the performance for low frequency ones.
[1]
Lior Wolf,et al.
Using the Output Embedding to Improve Language Models
,
2016,
EACL.
[2]
Alexei Baevski,et al.
Adaptive Input Representations for Neural Language Modeling
,
2018,
ICLR.
[3]
Kaiming He,et al.
Momentum Contrast for Unsupervised Visual Representation Learning
,
2019,
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[4]
Moustapha Cissé,et al.
Efficient softmax approximation for GPUs
,
2016,
ICML.
[5]
Omer Levy,et al.
Generalization through Memorization: Nearest Neighbor Language Models
,
2020,
ICLR.
[6]
Richard Socher,et al.
Regularizing and Optimizing LSTM Language Models
,
2017,
ICLR.