INEX 2007 Evaluation Measures

This paper describes the official measures of retrieval effectiveness that are employed for the Ad Hoc Track at INEX 2007. Whereas in earlier years all, but only, XML elements could be retrieved, the result format has been liberalized to arbitrary passages. In response, the INEX 2007 measures are based on the amount of highlighted text retrieved, leading to natural extensions of the well-established measures of precision and recall. The following measures are defined: The Focused Task is evaluated by interpolated precision at 1% recall (iP[0.01]) in terms of the highlighted text retrieved. The Relevant in Context Task is evaluated by mean average generalized precision (MAgP) where the generalized score per article is based on the retrieved highlighted text. The Best in Context Task is also evaluated by mean average generalized precision (MAgP) but here the generalized score per article is based on the distance to the assessor's best-entry point.

[1]  Gabriella Kazai,et al.  eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval , 2006, TOIS.

[2]  Djoerd Hiemstra,et al.  The Simplest Evaluation Measures for XML Information Retrieval that Could Possibly Work , 2005 .

[3]  Gabriella Kazai,et al.  INEX 2006 Evaluation Measures , 2006, INEX.

[4]  James Allan,et al.  HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents , 2003, TREC.

[5]  Marti A. Hearst,et al.  TREC 2007 Genomics Track Overview , 2007, TREC.

[6]  Mounia Lalmas,et al.  Overview of INEX 2004 , 2004, INEX.

[7]  Andrew Trotman,et al.  Proceedings of the SIGIR 2007 Workshop on Focused Retrieval: held in Amsterdam, The Netherlands, 27 July 2007 , 2008 .

[8]  Jaap Kamps,et al.  Evaluating relevant in context: document retrieval with a twist , 2007, SIGIR.

[9]  Andrew Trotman,et al.  INEX 2005 guidelines for topic development , 2005 .

[10]  Gabriella Kazai.  Initiative for the Evaluation of XML Retrieval , 2009 .

[11]  Hinrich Schütze,et al.  Introduction to Information Retrieval: Evaluation in information retrieval , 2008 .

[12]  Gabriella Kazai,et al.  Report of the INEX 2003 metrics working group , 2014 .

[13]  Gabriella Kazai,et al.  Advances in XML Information Retrieval and Evaluation, 4th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2005, Dagstuhl Castle, Germany, November 28-30, 2005, Revised Selected Papers , 2006, INEX.

[14]  Gabriella Kazai,et al.  Overview of the Initiative for the Evaluation of XML retrieval (INEX) 2002 , 2002, INEX Workshop.

[15]  Ludovic Denoyer,et al.  The XML Wikipedia Corpus , 2006 .

[16]  Mounia Lalmas,et al.  Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation , 2006, CIKM '06.

[17]  Andrew Trotman,et al.  Overview of the INEX 2007 Ad Hoc Track , 2008, INEX.

[18]  Ludovic Denoyer,et al.  The Wikipedia XML Corpus , 2006, INEX.

[19]  Andrew Trotman,et al.  Report on the SIGIR 2006 workshop on XML element retrieval methodology , 2006, SIGF.

[20]  Charles L. A. Clarke,et al.  INEX 2006 retrieval task and result submission specification , 2006 .

[21]  Gabriella Kazai,et al.  Advances in XML Information Retrieval and Evaluation: 4th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2005, Dagstuhl ... Papers (Lecture Notes in Computer Science) , 2006 .

[22]  Mounia Lalmas,et al.  Advances in XML Information Retrieval: Third International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2004, Dagstuhl Castle, ... 2004 (Lecture Notes in Computer Science) , 2005 .

[23]  Andrew Trotman,et al.  Passage Retrieval and other XML-Retrieval Tasks , 2006, SIGIR 2006.

[24]  Andrew Trotman,et al.  Report on the SIGIR 2007 workshop on focused retrieval , 2007, SIGF.

[25]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Novelty Track. , 2005 .

[26]  James Allan,et al.  Passage Retrieval and Evaluation , 2005 .

[27]  Mounia Lalmas,et al.  Advances in XML Information Retrieval, Third International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2004, Dagstuhl Castle, Germany, December 6-8, 2004, Revised Selected Papers , 2005, INEX.

[28]  James A. Thom,et al.  HiXEval: Highlighting XML Retrieval Evaluation , 2005, INEX.

[29]  Gabriella Kazai.  INitiative for the Evaluation of XML Retrieval , 2009, Encyclopedia of Database Systems.

[30]  Andrew Trotman,et al.  Comparative Evaluation of XML Information Retrieval Systems: 5th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2006 Dagstuhl Castle, Germany, December 17-20, 2006 Revised and Selected Papers , 2005 .

[31]  Gabriella Kazai,et al.  INEX 2005 Evaluation Measures , 2005, INEX.

[32]  James A. Thom,et al.  Evaluating Focused Retrieval Tasks , 2007 .

[33]  Gabriella Kazai,et al.  Overview of INEX 2005 , 2005, INEX.

[34]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[35]  Benjamin Piwowarski,et al.  Measurement, Theory , 2022 .