Introspection unit in memory network: Learning to generalize inference in OOV scenarios

Abstract Inference in natural language processing (NLP) is a tough task. Although plenty of models have been proposed in recent years, they are usually restricted to infer within a limited vocabulary or handcrafted training templates. In this paper, we propose the introspection unit (IU), a new neural module which can be incorporated with memory networks to deal with inference tasks in out of vocabulary (OOV) and rare named entities (RNEs) scenarios. Specifically, when encountering a new word, IU compares its part-of-speech context with the training dataset to extract a similar sample, and then embeds the new word into a target position to construct a simulated sample. The target position is located by the result of part-of-speech tagging. Finally, using the simulated sample, IU helps memory networks to learn the context and characteristic of the new word. In experiments, we evaluate the effectiveness of IU with the memory network on four inference datasets: a name OOV dataset, a place OOV dataset, a more challenging synthetical mixture OOV dataset and a realistic dialogue dataset. The experimental results demonstrate that IU effectively generalizes the inference ability of memory networks to OOV scenarios and improves the inference accuracies significantly. Furthermore, we visualize both the introspection process and the effect of IU in word embeddings and memories.