CLEF 2008: Ad Hoc Track Overview

We describe the objectives and organization of the CLEF 2008 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was changed considerably this year with the introduction of tasks with new document collections consisting of (i) library catalog records derived from The European Library, and (ii) and non-European language data, plus a task offering the chance to test retrieval with word sense disambiguated data. The track was thus structured in three distinct streams denominated: TEL@CLEF, Persian@CLEF and Robust WSD. The results obtained for each task are presented and statistical analyses are given.

[1]  Carol Peters,et al.  Advances in Multilingual and Multimodal Information Retrieval, 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers , 2008, CLEF.

[2]  Giorgio Maria Di Nunzio,et al.  A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns , 2007, EVIA@NTCIR.

[3]  Jacques Savoy,et al.  Stemming Approaches for East European Languages , 2008, CLEF.

[4]  Annalina Caputo,et al.  SENSE: SEmantic N-levels Search Engine at CLEF2008 Ad Hoc Robust-WSD Track , 2008, CLEF.

[5]  Marco Gonzalez,et al.  The PUCRS-PLN Group Participation at CLEF 2006 , 2006, CLEF.

[6]  Stephen Tomlinson.  German, French, English and Persian Retrieval Experiments at CLEF 2009 , 2008, CLEF.

[7]  Atelach Alemu Argaw.  Amharic-English Information Retrieval with Pseudo Relevance Feedback , 2007, CLEF.

[8]  Viviane Pereira Moreira,et al.  A Study on the use of Stemming for Monolingual Ad-Hoc Portuguese Information Retrieval , 2006, CLEF.

[9]  Masoud Rahgozar,et al.  Cross Language Experiments at Persian@CLEF 2008 , 2008, CLEF.

[10]  Viviane Pereira Moreira,et al.  UFRGS@CLEF2008: Using Association Rules for Cross-Language Information Retrieval , 2008, CLEF.

[11]  Miguel Ángel García Cumbreras,et al.  SINAI at Robust WSD Task @ CLEF 2008: When WSD is a Good Idea for Information Retrieval tasks? , 2008, CLEF.

[12]  Djoerd Hiemstra,et al.  WikiTranslate: Query Translation for Cross-lingual Information Retrieval using only Wikipedia , 2008, CLEF.

[13]  Carol Peters,et al.  Cross-Language Evaluation Forum: Objectives, Results, Achievements , 2004, Information Retrieval.

[14]  Annalina Caputo,et al.  UNIBA-SENSE at CLEF 2008: SEmantic N-levels Search Engine , 2008, CLEF.

[15]  Ray R. Larson.  Logistic Regression for Metadata: Cheshire takes on AdHoc-TEL , 2008, CLEF.

[16]  Jean-Michel Renders,et al.  XRCE's Participation to CLEF 2008 Ad-Hoc Track , 2008, CLEF.

[17]  Philipp Cimiano,et al.  Cross-language Information Retrieval with Explicit Semantic Analysis , 2008, CLEF.

[18]  Hwee Tou Ng,et al.  NUS-PT: Exploiting Parallel Texts for Word Sense Disambiguation in the English All-Words Tasks , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[19]  Jacques Savoy.  Why do successful search systems fail for some topics , 2007, SAC '07.

[20]  Piek T. J. M. Vossen,et al.  SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[21]  Farhad Oroumchian,et al.  Improving Persian Information Retrieval Systems Using Stemming and Part of Speech Tagging , 2008, CLEF.

[22]  Jean Tague-Sutcliffe,et al.  The Pragmatics of Information Retrieval Experimentation Revisited , 1997, Inf. Process. Manag..

[23]  Sudeshna Sarkar,et al.  Bengali and Hindi to English Cross-language Text Retrieval under Limited Resources , 2007, CLEF.

[24]  Stephen E. Robertson,et al.  On GMAP: and other transformations , 2006, CIKM '06.

[25]  Peter Willett,et al.  Readings in information retrieval , 1997 .

[26]  Aline Villavicencio,et al.  Indexing multiword expressions for information retrieval , 2008 .

[27]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Robust Retrieval Track , 2004 .

[28]  E. Ziegel.  Introduction to the Theory and Practice of Econometrics , 1989 .

[29]  W. J. Conover,et al.  Practical Nonparametric Statistics , 1972 .

[30]  Farhad Oroumchian,et al.  Using Part of Speech Tagging in Persian Information Retrieval , 2008, CLEF.

[31]  Maximilian Eibl,et al.  CLEF 2008 Ad-Hoc Track: On-line Processing Experiments with Xtrieval , 2008, CLEF.

[32]  Vasudeva Varma,et al.  Hindi and Telugu to English Cross Language Information Retrieval at CLEF 2006 , 2006, CLEF.

[33]  C. Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science) , 2005 .

[34]  Vasudeva Varma,et al.  Oromo-English Information Retrieval Experiments at CLEF 2007 , 2007, CLEF.

[35]  Carol Peters,et al.  CLEF 2003 Methodology and Metrics , 2003, CLEF.

[36]  Hugo Zaragoza,et al.  Query Clauses and Term Independence , 2008, CLEF.

[37]  Bogdan Sacaleanu,et al.  Working Notes for the CLEF 2008 Workshop , 2008 .

[38]  Carol Peters,et al.  CLEF 2005: Ad Hoc Track Overview , 2005, CLEF.

[39]  Mark Sanderson,et al.  Information retrieval system evaluation: effort, sensitivity, and reliability , 2005, SIGIR '05.

[40]  Dong Zhou,et al.  Ambiguity and Unknown Term Translation in CLIR , 2007, CLEF.

[41]  José Luis Borbinha,et al.  Technical University of Lisbon CLEF 2008 Submission: TEL@CLEF Monolingual Task , 2008, CLEF.

[42]  Hugo Zaragoza,et al.  UCM-Y!R at CLEF 2008 Robust and WSD tasks , 2008, CLEF.

[43]  José M. Perea-Ortega,et al.  Evaluating word sense disambiguation tools for information retrieval task , 2008 .

[44]  Claire Fautsch,et al.  UniNE at CLEF 2008: TEL, Persian and Robust IR , 2008, CLEF.

[45]  Miguel Ángel García Cumbreras,et al.  SINAI at CLEF 2006 Ad Hoc Robust Multilingual Track: Query Expansion using the Google Search Engine , 2006, CLEF.

[46]  Aline Villavicencio,et al.  UFRGS@CLEF2008: Indexing Multiword Expressions for Information Retrieval , 2008, CLEF.

[47]  Mirna Adriani,et al.  Evaluating Language Resources for English-Indonesian CLIR , 2006, CLEF.

[48]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[49]  Miguel Ángel García Cumbreras,et al.  Evaluating Word Sense Disambiguation Tools for Information Retrieval Task , 2008, CLEF.

[50]  Mirna Adriani,et al.  Evaluating Language Resources for CLEF 2007 , 2007, CLEF.

[51]  Jean-Michel Renders,et al.  Multi-language Models and Meta-dictionary Adaptation for Accessing Multilingual Digital Libraries , 2008, CLEF.

[52]  Luca Dini,et al.  CACAO Project at the TEL@CLEF 2009 Task , 2009, CLEF.

[53]  Eneko Agirre,et al.  UBC-ALM: Combining k-NN with SVD for WSD , 2007, SemEval@ACL.

[54]  Pavel Pecina,et al.  Charles University at CLEF 2007 Ad-Hoc Track , 2007, CLEF.

[55]  Sivaji Bandyopadhyay,et al.  Bengali, Hindi and Telugu to English Ad-hoc Bilingual Task at CLEF 2007 , 2007, CLEF.

[56]  Jacques Savoy,et al.  UniNE at CLEF 2006: Experiments with Monolingual, Bilingual, Domain-Specific and Robust Retrieval , 2006, CLEF.

[57]  C. Schönwiese,et al.  Overview of Results , 1997 .

[58]  Maria das Graças Volpe Nunes,et al.  Using Noun Phrases for Local Analysis in Automatic Query Expansion , 2006, CLEF.

[59]  Farhad Oroumchian,et al.  Fusion of Retrieval Models at CLEF 2008 Ad-Hoc Persian Track , 2008, CLEF.

[60]  Fernando Llopis,et al.  IRn in the CLEF Robust WSD Task 2008 , 2008, CLEF.

[61]  András A. Benczúr,et al.  Performing Cross-Language Retrieval with Wikipedia , 2007, CLEF.

[62]  Ángel F. Zazo Rodríguez,et al.  REINA at CLEF 2006 Robust Task: Local Query Expansion Using Term Windows for Robust Retrieval , 2006, CLEF.

[63]  Amir Hossein Jadidinejad,et al.  Investigation on Application of Local Cluster Analysis and Part of Speech Tagging on Persian Text , 2008, CLEF.

[64]  Stephen Tomlinson.  Sampling Precision to Depth 10000 at CLEF 2008 , 2008, CLEF.

[65]  Ellen M. Voorhees,et al.  The effect of topic set size on retrieval experiment error , 2002, SIGIR '02.

[66]  Carol Peters,et al.  The impact of evaluation on multilingual text retrieval , 2005, SIGIR '05.

[67]  Ludek Müller,et al.  Czech Monolingual Information Retrieval Using Off-The-Shelf Components - the University of West Bohemia at CLEF 2007 Ad-Hoc track , 2007, CLEF.

[68]  Giorgio Maria Di Nunzio,et al.  The Importance of Scientific Data Curation for Evaluation Campaigns , 2007, DELOS.

[69]  Gilles Falquet,et al.  Analysis of Word Sense Disambiguation-Based Information Retrieval , 2008, CLEF.

[70]  Paul McNamee.  JHU Ad Hoc Experiments at CLEF 2008 , 2008, CLEF.

[71]  Carol Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems , 2003, Lecture Notes in Computer Science.

[72]  Kurt Bilde,et al.  En forskningsartikel: This is a test , 2007 .

[73]  José Luis Borbinha,et al.  Experiments on a Multinomial Language Model versus Lucene's Off-the-Shelf Ranking Scheme and Rocchio Query Expansion (TEL@CLEF Monolingual Task) , 2008, CLEF.

[74]  Gilles Falquet,et al.  UNIGE Experiments on Robust Word Sense Disambiguation , 2008, CLEF.

[75]  A. Kumaran,et al.  Cross-Lingual Information Retrieval System for Indian Languages , 2008, IJCNLP.

[76]  Maximilian Eibl,et al.  CLEF 2008 Ad-Hoc Track: Comparing and Combining Different IR Approaches , 2008, CLEF.

[77]  Ellen M. Voorhees,et al.  The TREC robust retrieval track , 2005, SIGF.

[78]  Stephen Tomlinson,et al.  Comparing the Robustness of Expansion Techniques and Retrieval Measures , 2006, CLEF.

[79]  Arantxa Otegi,et al.  IXA at CLEF 2008 Robust-WSD Task: using Word Sense Disambiguation for (Cross Lingual) Information Retrieval , 2008, CLEF.

[80]  Pushpak Bhattacharyya,et al.  Hindi and Marathi to English Cross Language Information Retrieval , 2007, IJCNLP.

[81]  Claire Fautsch,et al.  UniNE at CLEF 2008: TEL, and Persian IR , 2008, CLEF.

[82]  Fernando Llopis,et al.  Applying Query Expansion Techniques to Ad Hoc Monolingual tasks with the IR-n system , 2007, CLEF.

[83]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[84]  Péter Halácsy.  Benefits of deep NLP-based Lemmatization for Information Retrieval , 2006, CLEF.

[85]  Vasudeva Varma,et al.  IIIT Hyderabad at CLEF 2007 - Adhoc Indian Language CLIR Task , 2007, CLEF.

[86]  Giorgio Maria Di Nunzio,et al.  Scientific Data of an Evaluation Campaign: Do We Properly Deal With Them? , 2006, CLEF.