Overview of Information Retrieval Evaluation

An important property of information retrieval (IR) system performance is its effectiveness at finding and ranking relevant documents in response to a user query. Research and development in IR requires rapid evaluation of effectiveness in order to test new approaches. This chapter covers the test collections required to evaluate effectiveness as well as traditional and newer measures of effectiveness.

[1]  Donna K. Harman,et al.  The TREC Conferences , 1997, HIM.

[2]  Ian Ruthven,et al.  Introduction to the special issue on evaluating interactive information retrieval systems , 2008, Inf. Process. Manag..

[3]  Stephen E. Robertson,et al.  The TREC-8 Filtering Track Final Report , 1999, TREC.

[4]  Donna K. Harman,et al.  Overview of the TREC 2002 Novelty Track , 2002, TREC.

[5]  C. W. Cleverdon,et al.  The testing of index language devices , 1997 .

[6]  K. Sparck Jones,et al.  INFORMATION RETRIEVAL TEST COLLECTIONS , 1976 .

[7]  Michael E. Lesk,et al.  Computer Evaluation of Indexing and Text Processing , 1968, JACM.

[8]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[9]  Charles L. A. Clarke,et al.  Efficient construction of large test collections , 1998, SIGIR '98.

[10]  James Allan,et al.  A comparison of statistical significance tests for information retrieval evaluation , 2007, CIKM '07.

[11]  Peter Willett,et al.  Readings in information retrieval , 1997 .

[12]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[13]  Ellen M. Voorhees,et al.  Bias and the limits of pooling , 2006, SIGIR '06.

[14]  Ophir Frieder,et al.  Using manually-built web directories for automatic evaluation of known-item retrieval , 2003, SIGIR.

[15]  Xiangji Huang,et al.  Overview of the TREC 2011 Chemical IR Track , 2009, TREC.

[16]  James Allan,et al.  Minimal test collections for retrieval evaluation , 2006, SIGIR.

[17]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[18]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[19]  Mounia Lalmas,et al.  SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval , 2006 .