An Overview of Evaluation Methods in TREC Ad Hoc Information Retrieval and TREC Question Answering

This chapter gives an overview of the current evaluation strategies and problems in the fields of information retrieval (IR) and question answering (QA), as instan- tiated in the Text Retrieval Conference (TREC). Whereas IR has a long tradition as a task, QA is a relatively new task which had to quickly develop its evaluation metrics, based on experiences gained in IR. This chapter will contrast the two tasks, their difficulties, and their evaluation metrics. We will end this chapter by pointing out limitations of the current evaluation strategies and potential future developments.

[1]  Donna K. Harman.  The First Text REtrieval Conference (TREC-1), Rockville, MD, USA, 4-6 November 1992 , 1993, Inf. Process. Manag..

[2]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[3]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[4]  Margaret King,et al.  Evaluating natural language processing systems , 1996, CACM.

[5]  James P. Callan,et al.  Passage-level evidence in document retrieval , 1994, SIGIR '94.

[6]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[7]  Karen Spärck Jones.  Reflections on TREC , 1995, Inf. Process. Manag..

[8]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[9]  Donna Harman,et al.  The First Text REtrieval Conference (TREC-1) , 1993 .

[10]  Ellen M. Voorhees,et al.  Building a question answering test collection , 2000, SIGIR '00.

[11]  K. Sparck Jones,et al.  INFORMATION RETRIEVAL TEST COLLECTIONS , 1976 .

[12]  Julia Galliers,et al.  Evaluating natural language processing systems , 1995 .

[13]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[14]  Sanda M. Harabagiu,et al.  Answer Mining by Combining Extraction Techniques with Abductive Reasoning , 2003, Text Retrieval Conference.

[15]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Novelty Track. , 2005 .

[16]  Karen Spärck Jones.  Further reflections on TREC , 2000, Inf. Process. Manag..