Query Performance Prediction: Evaluation Contrasted with Effectiveness

Query performance predictors are commonly evaluated by reporting correlation coefficients to denote how well the methods perform at predicting the retrieval performance of a set of queries. Despite the amount of research dedicated to this area, one aspect remains neglected: how strong does the correlation need to be in order to realize an improvement in retrieval effectiveness in an operational setting? We address this issue in the context of two settings: Selective Query Expansion and Meta-Search. In an empirical study, we control the quality of a predictor in order to examine how the strength of the correlation achieved, affects the effectiveness of an adaptive retrieval system. The results of this study show that many existing predictors fail to achieve a correlation strong enough to reliably improve the retrieval effectiveness in the Selective Query Expansion as well as the Meta-Search setting.

[1]  Shengli Wu,et al.  Data fusion with estimated weights , 2002, CIKM '02.

[2]  Claudio Carpineto,et al.  Query Difficulty, Robustness, and Selective Application of Query Expansion , 2004, ECIR.

[3]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[4]  Ingemar J. Cox,et al.  On ranking the effectiveness of searches , 2006, SIGIR.

[5]  W. Bruce Croft,et al.  Query expansion using local and global document analysis , 1996, SIGIR '96.

[6]  Leif Azzopardi,et al.  When is query performance prediction effective? , 2009, SIGIR.

[7]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[8]  Chris Buckley,et al.  Improving automatic query expansion , 1998, SIGIR '98.

[9]  W. Bruce Croft,et al.  A framework for selective query expansion , 2004, CIKM '04.

[10]  K. Kwok,et al.  An Attempt to Identify Weakest and Strongest Queries , 2005 .

[11]  Elad Yom-Tov,et al.  Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval , 2005, SIGIR '05.

[12]  Eitan Farchi,et al.  Automatic query wefinement using lexical affinities with maximal information gain , 2002, SIGIR '02.

[13]  Fernando Diaz,et al.  Performance prediction using spatial autocorrelation , 2007, SIGIR.

[14]  W. Bruce Croft,et al.  Relevance-Based Language Models , 2001, SIGIR '01.

[15]  W. Bruce Croft,et al.  Improving the effectiveness of information retrieval with local context analysis , 2000, TOIS.

[16]  Falk Scholer,et al.  Effective Pre-retrieval Query Performance Prediction Using Similarity and Variability Evidence , 2008, ECIR.

[17]  W. Bruce Croft,et al.  Query performance prediction in web search environments , 2007, SIGIR.

[18]  Iadh Ounis,et al.  Inferring Query Performance Using Pre-retrieval Predictors , 2004, SPIRE.

[19]  Carmel Domshlak,et al.  Towards robust query expansion: model selection in the language modeling framework , 2007, SIGIR.

[20]  Wojciech Rytter,et al.  Extracting Powers and Periods in a String from Its Runs Structure , 2010, SPIRE.

[21]  Justin Zobel,et al.  When query expansion fails , 2003, SIGIR '03.

[22]  W. Bruce Croft,et al.  Ranking robustness: a novel framework to predict query performance , 2006, CIKM '06.

[23]  Iadh Ounis,et al.  Combining fields for query expansion and adaptive query expansion , 2007, Inf. Process. Manag..