Modeling and Predicting Term Mismatch for Full-Text Retrieval

The probability that a term appears in a relevant document is a fundamental quantity in the theory of probabilistic information retrieval, however prior research provided few clues about how to estimate it reliably. Since this probability measures how likely it is that a term has to appear in a document in order for the document to be relevant, in this thesis, it is called term necessity. Equivalently, it is also the proportion of relevant documents that contain the term, thus measures term recall, or the complement of term mismatch. This thesis uses exploratory data analysis to identify common reasons that user-specified query terms fail to match relevant documents, develops features correlated with each reason, and integrates them into a model that can be trained from data. The resulting term necessity predictions can be used as term weights in state-of-the-art retrieval models to improve retrieval accuracy substantially. Feature-based necessity prediction also supports diagnosis and improvement of query components. The thesis research will develop several forms of diagnosis and intervention. The simplest form is interactive feedback in which potential problems with query components are identified for a person to fix. More nuanced approaches to automatic formulations of structured queries are based on interventions that address different causes of term mismatch. For example, removing unnecessary terms, expanding the terms that are likely to mismatch, and weighting term disjunctions after query expansion. Improved weighting of structured query components also provides a new approach to addressing the field-length biases that persist in state-of-the-art retrieval models for structured documents. Collectively, these interventions leverage term necessity predictions to address a variety of common problems related to formation of effective queries.