Dynamic Cutoff Prediction in Multi-Stage Retrieval Systems

Modern multi-stage retrieval systems are comprised of a candidate generation stage followed by one or more reranking stages. In such an architecture, the quality of the final ranked list may not be sensitive to the quality of the initial candidate pool, especially in terms of early precision. This provides several opportunities to increase retrieval efficiency without significantly sacrificing effectiveness. In this paper, we explore a new approach to dynamically predicting the size of an initial result set in the candidate generation stage, which can directly affect the overall efficiency and effectiveness of the entire system. Previous work exploring this tradeoff has focused on global parameter settings that apply to all queries, even though optimal settings vary across queries. In contrast, we propose a technique that makes a parameter prediction to maximize efficiency within an effectiveness envelope on a per query basis, using only static pre-retrieval features. Experimental results show that substantial efficiency gains are achievable. In addition, our framework provides a versatile tool that can be used to estimate the effectiveness-efficiency tradeoffs that are possible before selecting and tuning algorithms to make machine-learned predictions.

[1]  J. Shane Culpepper,et al.  Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments , 2015, Information Retrieval Journal.

[2]  Jimmy J. Lin,et al.  Fast candidate generation for two-phase document ranking: postings list intersection with bloom filters , 2012, CIKM.

[3]  Jimmy J. Lin,et al.  Document vector representations for feature extraction in multi-stage document ranking , 2013, Information Retrieval.

[4]  Tao Qin,et al.  LETOR: A benchmark collection for research on learning to rank for information retrieval , 2010, Information Retrieval.

[5]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[6]  J. Shane Culpepper,et al.  Exploring the magic of WAND , 2013, ADCS.

[7]  Huimin Zhao,et al.  An extended tuning method for cost-sensitive regression and forecasting , 2011, Decis. Support Syst..

[8]  Craig MacDonald,et al.  Efficient and effective retrieval using selective pruning , 2013, WSDM.

[9]  Craig MacDonald,et al.  About learning models with multiple query-dependent features , 2013, TOIS.

[10]  Stephen E. Robertson,et al.  Microsoft Research at TREC 2009: Web and Relevance Feedback Track , 2009, TREC.

[11]  MoffatAlistair,et al.  Rank-biased precision for measurement of retrieval effectiveness , 2008 .

[12]  Pedro M. Domingos.  MetaCost: a general method for making classifiers cost-sensitive , 1999, KDD '99.

[13]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[14]  Rodrygo L. T. Santos,et al.  The whens and hows of learning to rank for web search , 2012, Information Retrieval.

[15]  Craig MacDonald,et al.  Learning to predict response times for online query scheduling , 2012, SIGIR '12.

[16]  Olivier Chapelle,et al.  Expected reciprocal rank for graded relevance , 2009, CIKM.

[17]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[18]  Christopher J. C. Burges,et al.  From RankNet to LambdaRank to LambdaMART: An Overview , 2010 .

[19]  Tie-Yan Liu,et al.  Learning to Rank for Information Retrieval , 2011 .

[20]  J. Shane Culpepper,et al.  Score-safe term-dependency processing with hybrid indexes , 2014, SIGIR.

[21]  Craig MacDonald,et al.  On the usefulness of query features for learning to rank , 2012, CIKM.

[22]  Seung-won Hwang,et al.  Delayed-Dynamic-Selective (DDS) Prediction for Reducing Extreme Tail Latency in Web Search , 2015, WSDM.

[23]  Yi Chang,et al.  Yahoo! Learning to Rank Challenge Overview , 2010, Yahoo! Learning to Rank Challenge.

[24]  Alistair Moffat,et al.  A similarity measure for indefinite rankings , 2010, TOIS.

[25]  J. Shane Culpepper,et al.  Efficient Location-Aware Web Search , 2015, ADCS.

[26]  Elad Yom-Tov,et al.  Estimating the query difficulty for information retrieval , 2010, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[27]  Jimmy J. Lin,et al.  Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures , 2013, SIGIR.

[28]  J. Shane Culpepper,et al.  The effect of pooling and evaluation depth on IR metrics , 2016, Information Retrieval Journal.

[29]  Berkant Barla Cambazoglu,et al.  Early exit optimizations for additive machine learned ranking systems , 2010, WSDM '10.

[30]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[31]  Alistair Moffat,et al.  Rank-biased precision for measurement of retrieval effectiveness , 2008, TOIS.

[32]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[33]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[34]  Yiqun Liu,et al.  Is learning to rank effective for Web search ? , 2009 .

[35]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[36]  Charles L. A. Clarke,et al.  A Family of Rank Similarity Measures Based on Maximized Effectiveness Difference , 2015, IEEE Transactions on Knowledge and Data Engineering.