Creating a Dutch testbed to evaluate the retrieval from textual databases

This paper describes the first large-scale evaluation of information retrieval systems using Dutch documents and queries. We describe in detail the characteristics of the Dutch test data, which is part of the official CLEF multilingual texttual database, and give an overview of the experimental results of companies and research institutions that participated in the first official Dutch CLEF experiments. Judging from these experiments, the handling of language-specific issues of Dutch, like for instance simple morphology and compound nouns, significantly improves the performance of information retrieval systems in many cases. Careful examination of the test collection shows that it serves as a reliable tool for the evaluation of information retrieval systems in the future.

[1]  Peter van der Weerd,et al.  First Experiments with CLEF , 2001, CLEF.

[2]  Douglas W. Oard.  The CLEF 2001 Interactive Track , 2001, CLEF.

[3]  Isabelle Moulinier,et al.  Thomson Legal and Regulatory at CLEF 2001: Monolingual and Bilingual Experiments , 2001, CLEF.

[4]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[5]  Stephen P. Harter,et al.  Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness , 1996, J. Am. Soc. Inf. Sci..

[6]  Stefano Mizzaro.  Relevance: the whole history , 1997 .

[7]  Brian Vickery,et al.  Techniques of information retrieval , 1970 .

[8]  Stephen Tomlinson.  Hummingbird's Fulcrum SearchServer at CLEF2001 , 2001, CLEF.

[9]  James Mayfield,et al.  JHU/APL Experiments at CLEF: Translation Resources and Score Normalization , 2001, CLEF.

[10]  Mirna Adriani.  English-Dutch CLIR Using Query Translation Techniques , 2001, CLEF.

[11]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[12]  Annius Groenink,et al.  Minimalistic Test Runs of the Eidetica Indexer , 2001, CLEF.

[13]  Wessel Kraaij,et al.  TNO at CLEF-2001: Comparing Translation Resources , 2001, CLEF.

[14]  Maarten de Rijke,et al.  The University of Amsterdam at CLEF 2003 , 2001, CLEF.

[15]  C. J. van Rijsbergen,et al.  Report on the need for and provision of an 'ideal' information retrieval test collection , 1975 .

[16]  Michael E. Lesk,et al.  Relevance assessments and retrieval system evaluation , 1968, Inf. Storage Retr..

[17]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[18]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[19]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[20]  K. Sparck Jones,et al.  INFORMATION RETRIEVAL TEST COLLECTIONS , 1976 .

[21]  Marvin Brünner,et al.  Some Terms are more Interchangeable than Others , 2001, CLEF.