Contextual word similarity and estimation from sparse data

Abstract In recent years there is much interest in word co-occurrence relations, such as n-grams, verb–object combinations, or co-occurrence within a limited context. This paper discusses how to estimate the likelihood of co-occurrences that do not occur in the training data. We present a method that makes local analogies between each specific unobserved co-occurrence and other co-occurrences that contain similar words. These analogies are based on the assumption that similar word co-occurrences have similar values of mutual information. Accordingly, the word similarity metric captures similarities between vectors of mutual information values. Our evaluation suggests that this method performs better than existing, frequency-based, smoothing methods, and may provide an alternative to class-based models. A background survey is included, covering issues of lexical co-occurrence, data sparseness and smoothing, word similarity and clustering, and mutual information.