CLEF 2007: Ad Hoc Track Overview

We describe the objectives and organization of the CLEF 2007 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was divided into two streams. The main stream offered mono- and bilingual tasks on target collections for central European languages (Bulgarian, Czech and Hungarian). Similarly to last year, a bilingual task that encouraged system testing with non-European languages against English documents was also offered; this year, particular attention was given to Indian languages. The second stream, designed for more experienced participants, offered mono- and bilingual "robust" tasks with the objective of privileging experiments which achieve good stable performance over all queries rather than high average performance. These experiments re-used CLEF test collections from previous years in three languages (English, French, and Portuguese). The performance achieved for each task is presented and discussed.

[1]  M. F. Fuller,et al.  Practical Nonparametric Statistics; Nonparametric Statistical Inference , 1973 .

[2]  András A. Benczúr,et al.  Performing Cross-Language Retrieval with Wikipedia , 2007, CLEF.

[3]  Ellen M. Voorhees,et al.  The TREC robust retrieval track , 2005, SIGF.

[4]  E. Ziegel.  Introduction to the Theory and Practice of Econometrics , 1989 .

[5]  Prasenjit Majumder,et al.  Bulgarian, Hungarian and Czech Stemming Using YASS , 2007, CLEF.

[6]  Dong Zhou,et al.  Ambiguity and Unknown Term Translation in CLIR , 2007, CLEF.

[7]  Mark Sanderson,et al.  Information retrieval system evaluation: effort, sensitivity, and reliability , 2005, SIGIR '05.

[8]  Jacques Savoy,et al.  Stemming Approaches for East European Languages , 2008, CLEF.

[9]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[10]  Miguel Ángel García Cumbreras,et al.  SINAI at CLEF Ad-Hoc Robust Track 2007: Applying Google Search Engine for Robust Cross-Lingual Retrieval , 2007, CLEF.

[11]  Sudeshna Sarkar,et al.  Bengali and Hindi to English Cross-language Text Retrieval under Limited Resources , 2007, CLEF.

[12]  Mirna Adriani,et al.  Indonesian-English Transitive Translation for Cross-Language Information Retrieval , 2007, CLEF.

[13]  Vasudeva Varma,et al.  Oromo-English Information Retrieval Experiments at CLEF 2007 , 2007, CLEF.

[14]  Ángel F. Zazo Rodríguez,et al.  Improving Robustness Using Query Expansion , 2007, CLEF.

[15]  Stephen Tomlinson.  Sampling Precision to Depth 10000: Evaluation Experiments at CLEF 2007 , 2007, CLEF.

[16]  Mirna Adriani,et al.  Evaluating Language Resources for CLEF 2007 , 2007, CLEF.

[17]  José Carlos González Cristóbal,et al.  MIRACLE Progress in Monolingual Information Retrieval at Ad-Hoc CLEF 2007 , 2008 .

[18]  András A. Benczúr,et al.  Cross-Language Retrieval with Wikipedia , 2008, CLEF.

[19]  Fernando Llopis,et al.  Applying Query Expansion Techniques to Ad Hoc Monolingual tasks with the IR-n system , 2007, CLEF.

[20]  Vasudeva Varma,et al.  IIIT Hyderabad at CLEF 2007 - Adhoc Indian Language CLIR Task , 2007, CLEF.

[21]  Atelach Alemu Argaw.  Amharic-English Information Retrieval with Pseudo Relevance Feedback , 2007, CLEF.

[22]  Sudeshna Sarkar,et al.  Bengali and Hindi to English CLIR Evaluation , 2007, IJCNLP.

[23]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[24]  Peter Willett,et al.  Readings in information retrieval , 1997 .

[25]  Carol Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems , 2003, Lecture Notes in Computer Science.

[26]  Kurt Bilde,et al.  En forskningsartikel: This is a test , 2007 .

[27]  Giorgio Maria Di Nunzio,et al.  How robust are multilingual information retrieval systems? , 2008, SAC '08.

[28]  Vasudeva Varma,et al.  Improving Recall for Hindi, Telugu, Oromo to English CLIR , 2007, CLEF.

[29]  A. Kumaran,et al.  Cross-Lingual Information Retrieval System for Indian Languages , 2008, IJCNLP.

[30]  W. J. Conover,et al.  Practical Nonparametric Statistics , 1972 .

[31]  Jacques Savoy.  Why do successful search systems fail for some topics , 2007, SAC '07.

[32]  Pushpak Bhattacharyya,et al.  Hindi and Marathi to English Cross Language Information Retrieval , 2007, IJCNLP.

[33]  Miguel A. García-Cumbreras,et al.  SINAI at CLEF Ad-Hoc Robust Track 2007: Applying Google Search Engine for Robust Cross-Lingual Retrieval , 2008 .

[34]  Dong Zhou,et al.  Disambiguation and Unknown Term Translation in Cross Language Information Retrieval , 2007, CLEF.

[35]  C. Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science) , 2005 .

[36]  Carol Peters,et al.  CLEF 2003 Methodology and Metrics , 2003, CLEF.

[37]  Carol Peters,et al.  CLEF 2005: Ad Hoc Track Overview , 2005, CLEF.

[38]  Manuel Vilares Ferro,et al.  English-to-French CLIR: A Knowledge-Light Approach through Character N-Grams Alignment , 2007, CLEF.

[39]  G. Judge,et al.  Introduction to the Theory and Practice , 1988 .

[40]  Sivaji Bandyopadhyay,et al.  Bengali, Hindi and Telugu to English Ad-hoc Bilingual Task at CLEF 2007 , 2007, CLEF.

[41]  C. Schönwiese,et al.  Overview of Results , 1997 .

[42]  Prasenjit Majumder,et al.  Hungarian and Czech Stemming using YASS , 2007, CLEF.

[43]  Pushpak Bhattacharyya,et al.  Hindi to English and Marathi to English Cross Language Information Retrieval Evaluation , 2008, CLEF.

[44]  Ludek Müller,et al.  Czech Monolingual Information Retrieval Using Off-The-Shelf Components - the University of West Bohemia at CLEF 2007 Ad-Hoc track , 2007, CLEF.

[45]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[46]  Stephen Tomlinson.  Sampling Precision to Depth 10000 at CLEF 2007 , 2007, CLEF.

[47]  Pavel Pecina,et al.  Charles University at CLEF 2007 Ad-Hoc Track , 2007, CLEF.