Expected Ratio of Relevant Units: A Measure for Structured Information Retrieval

Since the 60’s, evaluation has been a key problem for Information Retrieval (IR) systems and has been extensively discussed in the IR community. New IR paradigms, like Structured Information Retrieval (SIR), make classical evaluation measures inappropriate. A few tentative extensions to these measures have been proposed but are also inadequate. We propose in this paper a new measure which is a generalisation of recall. This measure takes into account the specificity of SIR, when elements to be retrieved are linked by structural relationships. We show an instantiation of this measure on the INEX database and present experiments to show how well it is adapted to SIR evaluation.

[1]  Peter Bollmann-Sdorra,et al.  Measurement-theoretical investigation of the MZ-metric , 1980, SIGIR '80.

[2]  Jaana Kekäläinen,et al.  Using graded relevance assessments in IR evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[3]  John A. Swets,et al.  Effectiveness of information retrieval methods , 1969 .

[4]  Gabriella Kazai,et al.  Overview of the Initiative for the Evaluation of XML retrieval (INEX) 2002 , 2002, INEX Workshop.

[5]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[6]  William S. Cooper,et al.  Some inconsistencies and misidentified modeling assumptions in probabilistic information retrieval , 1995, TOIS.

[7]  Mounia Lalmas,et al.  A Dempster-Shafer indexing for the focused retrieval of a hierarchically structured document space: Implementation and experiments on a web museum collection , 2000, RIAO.

[8]  Mohamed S. Kamel,et al.  Formal methods for evaluating information retrieval in hypertext systems , 1993, SIGDOC '93.

[9]  Gabriella Kazai.  INitiative for the Evaluation of XML Retrieval , 2009, Encyclopedia of Database Systems.

[10]  Vijay V. Raghavan,et al.  A critical investigation of recall and precision as measures of retrieval system performance , 1989, TOIS.

[11]  D. R. Swanson.  Historical note: information retrieval and the future of an illusion , 1997 .

[12]  Stefano Mizzaro,et al.  How many relevances in information retrieval? , 1998, Interact. Comput..

[13]  S. Pollock.  Measures for the comparison of information retrieval systems , 1968 .