Automated Component-Level Evaluation: Present and Future

Automated component-level evaluation of information retrieval (IR) is the main focus of this paper. We present a review of the current state of web-based and component-level evaluation. Based on these systems, propositions are made for a comprehensive framework for web service-based component-level IR system evaluation. The advantages of such an approach are considered, as well as the requirements for implementing it. Acceptance of such systems by researchers who develop components and systems is crucial for having an impact and requires that a clear benefit is demonstrated.

[1]  Carol Peters,et al.  Multilingual Information Access Evaluation I. Text Retrieval Experiments, 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers , 2010, CLEF.

[2]  Donna Harman,et al.  The First Text REtrieval Conference (TREC-1) , 1993 .

[3]  Alistair Moffat,et al.  Improvements that don't add up: ad-hoc retrieval results since 1998 , 2009, CIKM.

[4]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[5]  Henning Müller,et al.  A web-based evaluation system for CBIR , 2001, MULTIMEDIA '01.

[6]  Jianwu Wang,et al.  Kepler + Hadoop: a general architecture facilitating data-intensive applications in scientific workflow systems , 2009, WORKS '09.

[7]  Luis von Ahn.  Games with a Purpose , 2006, Computer.

[8]  Donna K. Harman,et al.  CLEF 2009: Grid@CLEF Pilot Track Overview , 2009, CLEF.

[9]  Donna K. Harman,et al.  Overview of the First Text REtrieval Conference (TREC-1) , 1992, TREC.

[10]  Stephen Robertson,et al.  The methodology of information retrieval experiment , 1981 .

[11]  K. Cohen,et al.  Overview of BioCreative II gene normalization , 2008, Genome Biology.

[12]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project: report on the testing and analysis of an investigation into the comparative efficiency of indexing systems , 1962 .

[13]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[14]  J. Stephen Downie,et al.  The music information retrieval evaluation exchange (2005-2007): A window into music information retrieval research , 2008 .

[15]  Noriko Kando,et al.  Overview of the NTCIR-7 ACLIA Tasks: Advanced Cross-Lingual Information Access , 2008, NTCIR.

[16]  A. Valencia,et al.  Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge , 2008, Genome Biology.

[17]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[18]  Marcel Worring,et al.  The challenge problem for automated detection of 101 semantic concepts in multimedia , 2006, MM '06.

[19]  Thomas Deselaers,et al.  The Visual Concept Detection Task in ImageCLEF 2008 , 2008, CLEF.

[20]  Thierry Pun,et al.  Content-based query of image databases: inspirations from text retrieval , 2000, Pattern Recognit. Lett..

[21]  Thierry Pun,et al.  A Web-based evaluation system for content-based image retrieval , 2001 .