Automatic Ranking of Information Retrieval Systems

Typical information retrieval system evaluation requires expensive manually-collected relevance judgments of documents, which are used to rank retrieval systems. Due to the high cost associated with collecting relevance judgments and the ever-growing scale of data to be searched in practice, ranking of retrieval systems using manual judgments is becoming less feasible. Methods to automatically rank systems in absence of judgments have been proposed to tackle this challenge. However, current techniques are still far from reaching the ranking achieved using manual judgments. I propose to advance research on automatic system ranking using supervised and unsupervised techniques.

[1]  Miles Efron,et al.  Query polyrepresentation for ranking retrieval systems without relevance judgments , 2010, J. Assoc. Inf. Sci. Technol..

[2]  Ben He,et al.  Training query filtering for semi-supervised learning to rank with pseudo labels , 2015, World Wide Web.

[3]  Anselm Spoerri,et al.  Using the structure of overlap between search results to rank retrieval systems without relevance judgments , 2007, Inf. Process. Manag..

[4]  James Allan,et al.  Minimal test collections for retrieval evaluation , 2006, SIGIR.

[5]  Josiane Mothe,et al.  Learning to Rank System Configurations , 2016, CIKM.

[6]  Stephen E. Robertson,et al.  A few good topics: Experiments in topic set reduction for retrieval evaluation , 2009, TOIS.

[7]  Stephen E. Robertson,et al.  Hits hits TREC: exploring IR evaluation results with network analysis , 2007, SIGIR.

[8]  W. Bruce Croft,et al.  Iterative Search using Query Aspects , 2016, CIKM.

[9]  Rabia Nuray-Turan,et al.  Automatic ranking of information retrieval systems using data fusion , 2006, Inf. Process. Manag..

[10]  Tamer Elsayed,et al.  Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging , 2017, Inf. Process. Manag..

[11]  Douglas W. Oard,et al.  Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization , 2014, ECIR.

[12]  Miles Efron,et al.  Using Multiple Query Aspects to Build Test Collections without Human Relevance Judgments , 2009, ECIR.

[13]  Shengli Wu,et al.  Methods for ranking information retrieval systems without relevance judgments , 2003, SAC '03.

[14]  Ian Soboroff,et al.  Ranking retrieval systems without relevance judgments , 2001, SIGIR '01.

[15]  Djoerd Hiemstra,et al.  A Case for Automatic System Evaluation , 2010, ECIR.

[16]  Ellen M. Voorhees,et al.  Bias and the limits of pooling for large collections , 2007, Information Retrieval.

[17]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..