A Language Modeling Framework for Selective Query Expansion

Abstract : Query expansion is a well-known technique that has been shown to improve average retrieval performance. This technique has not been used in many operational systems because of the fact that it can greatly degrade the performance of some individual queries. We show how comparison between language models of the unexpanded and expanded retrieval results can be used to predict when the expanded retrieval has strayed from the original sense of the query. In these cases, the unexpanded results are used while the expanded results are used in the remaining cases (where such straying is not detected). We evaluate this method and others on a wide variety of TREC collections and show how to automatically compute a decision threshold for a collection. We demonstrate the ability of the method to enhance the effectiveness and reliability of the query expansion technique in information retrieval.