A Unified Framework for Post-Retrieval Query-Performance Prediction

The query-performance prediction task is estimating the effectiveness of a search performed in response to a query in lack of relevance judgments. Post-retrieval predictors analyze the result list of top-retrieved documents. While many of these previously proposed predictors are supposedly based on different principles, we show that they can actually be derived from a novel unified prediction framework that we propose. The framework is based on using a pseudo effective and/or ineffective ranking as reference comparisons to the ranking at hand, the quality of which we want to predict. Empirical exploration provides support to the underlying principles, and potential merits, of our framework.

[1]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[2]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[3]  Javed A. Aslam,et al.  Query Hardness Estimation Using Jensen-Shannon Divergence Among Multiple Scoring Functions , 2007, ECIR.

[4]  Wojciech Rytter,et al.  Extracting Powers and Periods in a String from Its Runs Structure , 2010, SPIRE.

[5]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[6]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[7]  W. Bruce Croft,et al.  Ranking robustness: a novel framework to predict query performance , 2006, CIKM '06.

[8]  Fernando Diaz,et al.  UMass at TREC 2004: Novelty and HARD , 2004, TREC.

[9]  Djoerd Hiemstra,et al.  The Combination and Evaluation of Query Performance Prediction Methods , 2009, ECIR.

[10]  Ingemar J. Cox,et al.  On ranking the effectiveness of searches , 2006, SIGIR.

[11]  Oren Kurland,et al.  Predicting Query Performance by Query-Drift Estimation , 2009, TOIS.

[12]  Milad Shokouhi,et al.  Advances in Information Retrieval Theory, Second International Conference on the Theory of Information Retrieval, ICTIR 2009, Cambridge, UK, September 10-12, 2009, Proceedings , 2009, ICTIR.

[13]  W. Bruce Croft,et al.  Query performance prediction in web search environments , 2007, SIGIR.

[14]  S. Robertson.  The probability ranking principle in IR , 1997 .

[15]  Elad Yom-Tov,et al.  Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval , 2005, SIGIR '05.

[16]  Oren Kurland,et al.  Using statistical decision theory and relevance models for query-performance prediction , 2010, SIGIR.

[17]  W. Bruce Croft,et al.  Retrieval performance prediction and document quality , 2008 .

[18]  Elad Yom-Tov,et al.  Estimating the query difficulty for information retrieval , 2010, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[19]  Fernando Diaz,et al.  Performance prediction using spatial autocorrelation , 2007, SIGIR.

[20]  John D. Lafferty,et al.  A study of smoothing methods for language models applied to Ad Hoc information retrieval , 2001, SIGIR '01.

[21]  W. Bruce Croft,et al.  A general language model for information retrieval (poster abstract) , 1999, Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[22]  Iadh Ounis,et al.  Inferring Query Performance Using Pre-retrieval Predictors , 2004, SPIRE.

[23]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[24]  Elad Yom-Tov,et al.  What makes a query difficult? , 2006, SIGIR.

[25]  W. Bruce Croft,et al.  A Language Modeling Framework for Selective Query Expansion , 2004 .

[26]  Steven Garcia,et al.  RMIT University at TREC 2005: Terabyte and Robust Track , 2005, TREC.