Beyond Recall and Precision : A Full Framework for MIR System Evaluation

The Cranfield tests are perhaps the most well-known and often cited example of benchmarking of information retrieval systems. However, of the six criteria that Cleverdon identified as pertinent for analysis of information retrieval systems, only two, precision and recall, are typically investigated. We argue that the other criteria are also vitally important for advanced IR systems such as a music information retrieval (MIR) system. They should be modified and put into the appropriate framework for MIR systems. Furthermore, a systematic method of measuring all valid criteria should be devised. This paper considers similar attempts with other advanced IR systems, and suggests how to establish and measure the appropriate criteria for information retrieval systems to be used in conjunction with a Music Digital Library (MDL).

[1]  Kjell Lemström,et al.  SEMEX - An efficient Music Retrieval Prototype , 2000, ISMIR.

[2]  Clu-istos Foutsos,et al.  Fast subsequence matching in time-series databases , 1994, SIGMOD '94.

[3]  Haixun Wang,et al.  The S2-Tree : An Index Structure for Subsequence Matching of Spatial Objects , 2001, PAKDD.

[4]  Antonin Guttman,et al.  R-trees: a dynamic index structure for spatial searching , 1984, SIGMOD '84.

[5]  Aslib,et al.  The journal of documentation , 1945 .

[6]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[7]  Thomas Schreiber,et al.  EFFICIENT NEIGHBOR SEARCHING IN NONLINEAR TIME SERIES ANALYSIS , 1995 .

[8]  William R. Hersh,et al.  A task-oriented approach to information retrieval evaluation , 1996 .

[9]  Alfred Glossbrenner,et al.  Search engines for the World Wide Web , 1997 .

[10]  Jon Louis Bentley,et al.  K-d trees for semidynamic point sets , 1990, SCG '90.

[11]  Rainer Hoch,et al.  On the evaluation of document analysis components by recall, precision, and accuracy , 1999, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318).

[12]  Arbee L. P. Chen,et al.  Efficient Near Neighbor Searching Using Multi-Indexes for Content-Based Multimedia Data Retrieval , 2004, Multimedia Tools and Applications.

[13]  Jean Tague-Sutcliffe,et al.  Some Perspectives on the Evaluation of Information Retrieval Systems , 1996, J. Am. Soc. Inf. Sci..

[14]  Mark Sandler,et al.  Efficient Multidimensional Searching Routines , 2001, ISMIR.

[15]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[16]  Andrew W. Moore,et al.  Efficient memory-based learning for robot control , 1990 .

[17]  Peter N. Yianilos,et al.  Data structures and algorithms for nearest neighbor search in general metric spaces , 1993, SODA '93.

[18]  Jorma Tarhio,et al.  Searching monophonic patterns within polyphonic sources , 2000 .

[19]  Dragomir R. Radev.  Learning Correlations between Linguistic Indicators and Semantic Constraints: Reuse of Context-Dependent Descriptions of Entities , 1998, ACL.

[20]  Xavier Rodet,et al.  Automatic Characterisation of Musical Signals: Feature Extraction and Temporal Segmentation , 1999 .