Evaluating the effectiveness of content-oriented XML retrieval

Content-oriented XML retrieval approaches aim at a more focused retrieval strategy: Instead of retrieving whole documents, document components that are exhaustive to the information need while at the same time being as specific as possible should be retrieved. In this article, we show that the evaluation methods developed for standard retrieval must be modified in order to deal with the structure of XML documents. More precisely, the size and overlap of document components must be taken into account. For this purpose, we propose a new effectiveness metric based on the definition of a concept space defined upon the notions of exhaustiveness and specificity of a search result. We compare the results of this new metric by the results obtained with the official metric used in INEX, the evaluation initiative for content-oriented XML retrieval.

[1]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[2]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[3]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[4]  C. J. van Rijsbergen,et al.  Report on the need for and provision of an 'ideal' information retrieval test collection , 1975 .

[5]  K. Sparck Jones,et al.  INFORMATION RETRIEVAL TEST COLLECTIONS , 1976 .

[6]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[7]  C. J. van Rijsbergen,et al.  Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval , 1987, SIGIR 1987.

[8]  Vijay V. Raghavan,et al.  A critical investigation of recall and precision as measures of retrieval system performance , 1989, TOIS.

[9]  Donna K. Harman,et al.  Overview of the First Text REtrieval Conference (TREC-1) , 1992, TREC.

[10]  Donna Harman,et al.  The First Text REtrieval Conference (TREC-1) , 1993 .

[11]  Donna Harman,et al.  Overview of the First Text REtrieval Conference. , 1993, SIGIR 1993.

[12]  Yiyu Yao,et al.  On modeling information retrieval with probabilistic inference , 1995, TOIS.

[13]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[14]  Yves Chiaramella,et al.  A Model for Multimedia Information Retrieval , 1996 .

[15]  Edie Rasmussen,et al.  Evaluating interactive systems in TREC , 1996 .

[16]  Stephen E. Robertson,et al.  Evaluating Interactive Systems in TREC , 1996, J. Am. Soc. Inf. Sci..

[17]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[18]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[19]  Steven J. DeRose,et al.  XML Path Language (XPath) Version 1.0 , 1999 .

[20]  Peter Ingwersen,et al.  Dimensions of relevance , 2000, Inf. Process. Manag..

[21]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[22]  N. Fuhr.  PAN-Uncovering Plagiarism , Authorship , and Social Software Misuse ImageCLEF 2013-Cross Language Image Annotation and Retrieval INEX-INitiative for the Evaluation of XML retrieval , 2002 .

[23]  Carol Peters,et al.  Evaluation of Cross-Language Information Retrieval Systems , 2002, Lecture Notes in Computer Science.

[24]  Gabriella Kazai,et al.  Overview of the Initiative for the Evaluation of XML retrieval (INEX) 2002 , 2002, INEX Workshop.

[25]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[26]  Ellen M. Voorhees,et al.  The Tenth Text REtrieval Conference, TREC 2001 | NIST , 2002 .

[27]  Gabriella Kazai,et al.  Report of the INEX 2003 metrics working group , 2014 .

[28]  Gabriella Kazai,et al.  Construction of a Test Collection for the Focussed Retrieval of Structured Documents , 2003, ECIR.

[29]  Jun Adachi,et al.  Report from the NTCIR workshop 3 , 2004, SIGF.

[30]  Birger Larsen,et al.  The Interactive Track at INEX 2004 , 2004, INEX.

[31]  Gabriella Kazai,et al.  The overlap problem in content-oriented XML retrieval evaluation , 2004, SIGIR '04.

[32]  Mounia Lalmas,et al.  Providing consistent and exhaustive relevance assessments for XML retrieval evaluation , 2004, CIKM '04.

[33]  Mounia Lalmas,et al.  Report on the INEX 2003 workshop , 2004, SIGF.

[34]  David Hawking,et al.  Overview of the TREC 2004 Web Track , 2004, TREC.

[35]  Mounia Lalmas,et al.  Overview of INEX 2004 , 2004, INEX.

[36]  C. Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science) , 2005 .

[37]  Benjamin Piwowarski,et al.  Expected Ratio of Relevant Units: A Measure for Structured Information Retrieval , 2008 .