Notes on what to measure in INEX

This paper looks at a number of issues regarding the evaluation of XML retrieval. It aims to identify what the requirements on a measure of XML retrieval effectiveness are and how the actual evaluation methodology and aspects such as the relevance dimensions and the assessment procedure affect the evaluation. We examine various current and proposed metrics, how they fit the requirements and aim to give an explanation of what exactly they measure. A question we are attempting to address is: “Is there a single good measure of retrieval effectiveness for XML retrieval?”.

[1]  Gianni Amati,et al.  Probability models for information retrieval based on divergence from randomness , 2003 .

[2]  Gabriella Kazai,et al.  Construction of a Test Collection for the Focussed Retrieval of Structured Documents , 2003, ECIR.

[3]  Gabriella Kazai,et al.  Reliability Tests for the XCG and inex-2002 Metrics , 2004, INEX.

[4]  Mounia Lalmas,et al.  Best entry points for structured document retrieval - Part II: Types, usage and effectiveness , 2006, Inf. Process. Manag..

[5]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[6]  Gabriella Kazai,et al.  Overview of the Initiative for the Evaluation of XML retrieval (INEX) 2002 , 2002, INEX Workshop.

[7]  Charles L. A. Clarke,et al.  INEX 2006 retrieval task and result submission specification , 2006 .

[8]  Gabriella Kazai,et al.  The overlap problem in content-oriented XML retrieval evaluation , 2004, SIGIR '04.

[9]  Gabriella Kazai,et al.  Report of the INEX 2003 metrics working group , 2014 .

[10]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[11]  Birger Larsen,et al.  The Interactive Track at INEX 2004 , 2004, INEX.

[12]  Gabriella Kazai,et al.  Tolerance to irrelevance: a user-effort oriented evaluation of retrieval systems without predefined retrieval unit , 2004 .

[13]  Mounia Lalmas,et al.  Best entry points for structured document retrieval - Part I: Characteristics , 2006, Inf. Process. Manag..

[14]  Yiyu Yao,et al.  On modeling information retrieval with probabilistic inference , 1995, TOIS.

[15]  Gabriella Kazai,et al.  Evaluating the effectiveness of content-oriented XML retrieval , 2003 .

[16]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[17]  Eero Sormunen,et al.  Liberal relevance criteria of TREC -: counting on negligible documents? , 2002, SIGIR '02.

[18]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[19]  RaghavanVijay,et al.  A critical investigation of recall and precision as measures of retrieval system performance , 1989 .