CLEF 15th Birthday

2014 marks the 15th birthday for CLEF, an evaluation campaign activity which has applied the Cranfield evaluation paradigm to the testing of multilingual and multimodal information access systems in Europe. This paper provides a summary of the motivations which led to the establishment of CLEF, and a description of how it has evolved over the years, the major achievements, and what we see as the next challenges.

[1]  Nicola Ferro,et al.  DESIRE 2011: workshop on data infrastructurEs for supporting information retrieval evaluation , 2012, SIGF.

[2]  Julio Gonzalo,et al.  Overview of RepLab 2014: Author Profiling and Reputation Dimensions for Online Reputation Management , 2014, CLEF.

[3]  Piek T. J. M. Vossen,et al.  SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[4]  Martha Larson,et al.  Overview of VideoCLEF 2009: New Perspectives on Speech-based Multimedia Content Enrichment , 2009, CLEF.

[5]  Elaine G. Toms,et al.  Information Access Evaluation. Multilinguality, Multimodality, and Interaction , 2014 .

[6]  Rafael Berlanga Llavori,et al.  Semantic Discovery of Resources in Cloud-Based PACS/RIS Systems , 2013, CLEF.

[7]  Dietrich Rebholz-Schuhmann,et al.  Entity Recognition in Parallel Multi-lingual Biomedical Corpora: The CLEF-ER Laboratory Overview , 2013, CLEF.

[8]  Julio Gonzalo,et al.  Filter Keywords and Majority Class Strategies for Company Name Disambiguation in Twitter , 2011, CLEF.

[9]  M. de Rijke,et al.  Overview of WebCLEF 2006 , 2006, CLEF.

[10]  Shlomo Argamon,et al.  Overview of the International Authorship Identification Competition at PAN-2011 , 2011, CLEF.

[11]  Lori Lamel,et al.  Overview of QAST 2008 , 2008, CLEF.

[12]  Demetrios G. Glinos.  Discovering Similar Passages within Large Text Documents , 2014, CLEF.

[13]  Thomas Martin Deserno,et al.  The CLEF 2005 Cross-Language Image Retrieval Track , 2005, CLEF.

[14]  Giorgio Maria Di Nunzio,et al.  LogCLEF 2011 Multilingual Log File Analysis: Language Identification, Query Classification, and Success of a Query , 2011, CLEF.

[15]  Carol Peters,et al.  CLEF 2007: Ad Hoc Track Overview , 2008, CLEF.

[16]  Heng Ji,et al.  Analysis and Refinement of Cross-Lingual Entity Linking , 2012, CLEF.

[17]  C. Schönwiese,et al.  Overview of Results , 1997 .

[18]  Josef Steinberger,et al.  Using Parallel Corpora for Multilingual (Multi-document) Summarisation Evaluation , 2010, CLEF.

[19]  Martin Braschler,et al.  CLEF 2003 - Overview of Results , 2002, CLEF.

[20]  Douglas W. Oard,et al.  Building a Cross-Language Entity Linking Collection in Twenty-One Languages , 2011, CLEF.

[21]  Giorgio Maria Di Nunzio,et al.  Multilingual Log Analysis: LogCLEF , 2011, ECIR.

[22]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[23]  Benno Stein,et al.  Improving the Reproducibility of PAN's Shared Tasks: - Plagiarism Detection, Author Identification, and Author Profiling , 2014, CLEF.

[24]  Nils Pharo,et al.  Seven Years of INEX Interactive Retrieval Experiments - Lessons and Challenges , 2012, CLEF.

[25]  Gobinda G. Chowdhury,et al.  TREC: Experiment and Evaluation in Information Retrieval , 2007 .

[26]  Katja Hofmann,et al.  Validating Query Simulators: An Experiment Using Commercial Searches and Purchases , 2010, CLEF.

[27]  Sivaji Bandyopadhyay,et al.  Language Independent Query Focused Snippet Generation , 2012, CLEF.

[28]  Elena Cabrio,et al.  Multilingual Question Answering over Linked Data (QALD-3): Lab Overview , 2013, CLEF.

[29]  Roser Morante,et al.  QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation , 2013, CLEF.

[30]  Andrew Trotman,et al.  Overview of the INEX 2013 Snippet Retrieval Track , 2013, CLEF.

[31]  Ryen W. White,et al.  Overview of the CLEF-2005 Cross-Language Speech Retrieval Track , 2005, CLEF.

[32]  Ellen M. Voorhees,et al.  TREC: Continuing information retrieval's tradition of experimentation , 2007, CACM.

[33]  Maarten de Rijke,et al.  Overview of the CLEF 2004 Multilingual Question Answering Track , 2004, CLEF.

[34]  Martha Larson,et al.  Overview of VideoCLEF 2008: Automatic Generation of Topic-based Feeds for Dual Language Audio-Visual Content , 2008, CLEF.

[35]  Henning Müller,et al.  Overview of the CLEF 2009 Medical Image Retrieval Track , 2009, CLEF.

[36]  Carol Peters,et al.  Multilingual Information Access Evaluation I. Text Retrieval Experiments, 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Corfu, Greece, September 30 - October 2, 2009, Revised Selected Papers , 2010, CLEF.

[37]  Patrice Bellot,et al.  Overview of the INEX 2012 Tweet Contextualization Track , 2012, CLEF.

[38]  Gareth J. F. Jones,et al.  Simulation of Within-Session Query Variations Using a Text Segmentation Approach , 2011, CLEF.

[39]  Roser Morante,et al.  Overview of the QA4MRE Pilot Task: Annotating Modality and Negation for a Machine Reading Evaluation , 2011, CLEF.

[40]  Roberto Paredes,et al.  Overview of the ImageCLEF 2012 Scalable Web Image Annotation Task , 2012, CLEF.

[41]  Andreas Rauber,et al.  Mining Query Logs of USPTO Patent Examiners , 2013, CLEF.

[42]  Khalid Choukri,et al.  Information Filtering Evaluation: Overview of CLEF 2009 INFILE Track , 2009, CLEF.

[43]  Giorgio Maria Di Nunzio,et al.  A Proposal to Extend and Enrich the Scientific Data Curation of Evaluation Campaigns , 2007, EVIA@NTCIR.

[44]  Jorma Laaksonen,et al.  Overview of the ImageCLEF 2007 Object Retrieval Task , 2008, CLEF.

[45]  Mikko Kurimo,et al.  Morpho Challenge Evaluation Using a Linguistic Gold Standard , 2007, CLEF.

[46]  Andrea Esuli,et al.  Evaluating Information Extraction , 2010, CLEF.

[47]  Maarten de Rijke,et al.  The Multiple Language Question Answering Track at CLEF 2003 , 2003, CLEF.

[48]  Carol Peters,et al.  Evaluation of cross-language information retrieval systems : Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001, Darmstadt, Germany, September 3-4, 2001 : revised papers , 2002 .

[49]  Paolo Rosso,et al.  A New Corpus for the Evaluation of Arabic Intrinsic Plagiarism Detection , 2013, CLEF.

[50]  Henning Müller,et al.  Overview of the CLEF 2011 Medical Image Classification and Retrieval Tasks , 2011, CLEF.

[51]  Thomas Deselaers,et al.  Overview of the ImageCLEF 2006 Photographic Retrieval and Object Annotation Tasks , 2006, CLEF.

[52]  Julio Gonzalo,et al.  iCLEF 2006 Overview: Searching the Flickr WWW Photo-Sharing Repository , 2006, CLEF.

[53]  Hanna Suominen.  CLEFeHealth2012 - The CLEF 2012 Workshop on Cross-Language Evaluation of Methods, Applications, and Resources for eHealth Document Analysis , 2012, CLEF.

[54]  Julio Gonzalo,et al.  The CLEF 2002 Interactive Track , 2002, CLEF.

[55]  Benno Stein,et al.  Recent Trends in Digital Text Forensics and Its Evaluation - Plagiarism Detection, Author Identification, and Author Profiling , 2013, CLEF.

[56]  Djoerd Hiemstra,et al.  Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics , 2012, Lecture Notes in Computer Science.

[57]  Nicola Orio,et al.  Overview of MusiCLEF 2011 , 2011, CLEF.

[58]  Henning Müller,et al.  Overview of the ImageCLEFmed 2008 Medical Image Retrieval Task , 2008, CLEF.

[59]  Nicola Ferro,et al.  Cultural Heritage in CLEF (CHiC) , 2013 .

[60]  Henning Müller,et al.  Assessing the Scholarly Impact of ImageCLEF , 2011, CLEF.

[61]  Gareth J. F. Jones,et al.  Overview of the CLEF-2005 Cross-Language Speech Retrieval Track , 2005, CLEF.

[62]  Fredric C. Gey,et al.  ENSM-SE at CLEF 2006 : Fuzzy Proximity Method with an Adhoc Influence Function in Evaluation of Multilingual and Multi-modal Information Retrieval 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain , 2007 .

[63]  Allan Hanbury,et al.  CLEF-IP 2011: Retrieval in the Intellectual Property Domain , 2011, CLEF.

[64]  Gabriella Kazai,et al.  Overview of the INEX 2011 Books and Social Search Track , 2011, INEX.

[65]  Julio Gonzalo,et al.  Overview of RepLab 2012: Evaluating Online Reputation Management Systems , 2012, CLEF.

[66]  Julio Gonzalo,et al.  A Comparison of Evaluation Metrics for Document Filtering , 2011, CLEF.

[67]  Giorgio Maria Di Nunzio,et al.  LogCLEF 2009: the CLEF 2009 Multilingual Logfile Analysis Track Overview , 2009, CLEF.

[68]  M. Felisa Verdejo,et al.  Overview of the Answer Validation Exercise 2007 , 2006, CLEF.

[69]  Alan F. Smeaton,et al.  Multilingual and Multimodal Information Access Evaluation, International Conference of the Cross-Language Evaluation Forum, CLEF 2010, Padua, Italy, September 20-23, 2010. Proceedings , 2010, CLEF.

[70]  Vivien Petras,et al.  The Domain-Specific Track at CLEF 2008 , 2008, CLEF.

[71]  Stefanie Nowak,et al.  Overview of the CLEF 2009 Large Scale - Visual Concept Detection and Annotation Task , 2009, CLEF.

[72]  Carol Peters,et al.  CLEF 2006: Ad Hoc Track Overview , 2006, CLEF.

[73]  Harald Hammarström.  Automatic Annotation of Bibliographical References for Descriptive Language Materials , 2011, CLEF.

[74]  Miguel Cazorla,et al.  ImageCLEF 2014: Overview and Analysis of the Results , 2014, CLEF.

[75]  Giuseppe Santucci,et al.  Measuring and Analyzing the Scholarly Impact of Experimental Evaluation Initiatives , 2014, IRCDL.

[76]  Barbara Caputo,et al.  Overview of the CLEF 2009 Robot Vision Track , 2009, CLEF.

[77]  Albert N. Link,et al.  Economic impact assessment of NIST's text REtrieval conference (TREC) program. Final report , 2010 .

[78]  Nicola Ferro,et al.  Rethinking How to Extend Average Precision to Graded Relevance , 2014, CLEF.

[79]  Julio Gonzalo,et al.  Overview of iCLEF 2009: Exploring Search Behaviour in a Multilingual Folksonomy Environment , 2009, CLEF.

[80]  Peter Petros,et al.  Anatomy of the perineal membrane. , 2002, The Australian & New Zealand journal of obstetrics & gynaecology.

[81]  Achim Rettinger,et al.  A Comparative Evaluation of Cross-Lingual Text Annotation Techniques , 2013, CLEF.

[82]  Carol Peters,et al.  CLEF 2008: Ad Hoc Track Overview , 2008, CLEF.

[83]  Alexander Sachs,et al.  A Formative Evaluation of a Comprehensive Search System for Medical Professionals , 2013, CLEF.

[84]  Martin Potthast,et al.  Overview of the 1st International Competition on Wikipedia Vandalism Detection , 2010, CLEF.

[85]  Douglas W. Oard.  The CLEF 2001 Interactive Track , 2001, CLEF.

[86]  Allan Hanbury,et al.  Overview of CLEF-IP 2013 Lab - Information Retrieval in the Patent Domain , 2013, CLEF.

[87]  Donna K. Harman,et al.  CLEF 2009: Grid@CLEF Pilot Track Overview , 2009, CLEF.

[88]  Jeff Z. Pan,et al.  Making Test Corpora for Question Answering More Representative , 2014, CLEF.

[89]  Matthias Hagen,et al.  Overview of the 1st international competition on plagiarism detection , 2009 .

[90]  Nicola Ferro,et al.  DIRECTions: Design and Specification of an IR Evaluation Infrastructure , 2012, CLEF.

[91]  Nicola Ferro,et al.  Making it Easier to Discover, Re-Use and Understand Search Engine Experimental Evaluation Data , 2014, ERCIM News.

[92]  Theodora Tsikrika,et al.  The Wikipedia Image Retrieval Task , 2010, ImageCLEF.

[93]  Khalid Choukri,et al.  Overview of CLEF 2008 INFILE Pilot Track , 2008, CLEF.

[94]  Stefanie Nowak,et al.  Overview of the Photo Annotation Task in ImageCLEF@ICPR , 2010, ICPR Contests.

[95]  W. Bruce Croft,et al.  Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne , 2012, SIGF.

[96]  Allan Hanbury,et al.  An Information Retrieval Ontology for Information Retrieval Nanopublications , 2014, CLEF.

[97]  Maarten Marx,et al.  Overview of the INEX 2011 Data-Centric Track , 2011, INEX.

[98]  Eric SanJuan,et al.  A Turing Test to Evaluate a Complex Summarization Task , 2013, CLEF.

[99]  Nicola Ferro,et al.  Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns , 2009, ECDL.

[100]  Allan Hanbury,et al.  Automated Component-Level Evaluation: Present and Future , 2010, CLEF.

[101]  Mikko Kurimo,et al.  Overview of Morpho Challenge 2008 , 2008, CLEF.

[102]  Mark Sanderson,et al.  Diversity in Photo Retrieval: Overview of the ImageCLEFPhoto Task 2009 , 2009, CLEF.

[103]  Francisco J. Valverde-Albacete,et al.  A Proposal for New Evaluation Metrics and Result Visualization Technique for Sentiment Analysis Tasks , 2013, CLEF.

[104]  Alberto Barrón-Cedeño,et al.  Cross-Language High Similarity Search Using a Conceptual Thesaurus , 2012, CLEF.

[105]  Romain Rouvoy,et al.  Debugging with the Crowd: A Debug Recommendation System Based on StackOverflow , 2014, ERCIM News.

[106]  Fredric C. Gey,et al.  GeoCLEF 2008: the CLEF 2008 Cross-Language Geographic Information Retrieval Track Overview , 2008, CLEF.

[107]  Djoerd Hiemstra,et al.  MapReduce for Information Retrieval Evaluation: "Let's Quickly Test This on 12 TB of Data" , 2010, CLEF.

[108]  Bart Thomee,et al.  Overview of the ImageCLEF 2012 Flickr Photo Annotation and Retrieval Task , 2012, CLEF.

[109]  Arantxa Otegi,et al.  CLEF 2009 Ad Hoc Track Overview: Robust - WSD Task , 2009, CLEF.

[110]  Thomas Martin Deserno,et al.  Overview of the ImageCLEFmed 2007 Medical Retrieval and Medical Annotation Tasks , 2007, CLEF.

[111]  Carol Peters,et al.  Multilingual Information Retrieval , 2012, Springer Berlin Heidelberg.

[112]  Carol Peters,et al.  CLEF 2004: Ad Hoc Track Overview and Results Analysis , 2004, CLEF.

[113]  Julio Gonzalo,et al.  Overview of iCLEF 2008: Search Log Analysis for Multilingual Image Retrieval , 2008, CLEF.

[114]  Barbara Caputo,et al.  The Robot Vision Track at ImageCLEF 2010 , 2010, CLEF.

[115]  Walid Magdy,et al.  Examining the Robustness of Evaluation Metrics for Patent Retrieval with Incomplete Relevance Judgements , 2010, CLEF.

[116]  Paolo Rosso,et al.  Overview of QAST 2009 , 2009, CLEF.

[117]  Benno Stein,et al.  Overview of the 1th International Competition on Quality Flaw Prediction in Wikipedia , 2012, CLEF.

[118]  Martin Braschler,et al.  CLEF 2000 - Overview of Results , 2000, CLEF.

[119]  Paul Clough,et al.  A proposal for the CLEF Cross-Language Image Retrieval Track 2004 , 2004 .

[120]  Gareth J. F. Jones,et al.  A Case Study in Decompounding for Bengali Information Retrieval , 2013, CLEF.

[121]  Nazanin Dehghani,et al.  A Dependency-Inspired Semantic Evaluation of Machine Translation Systems , 2013, CLEF.

[122]  Giorgio Maria Di Nunzio,et al.  Scientific Data of an Evaluation Campaign: Do We Properly Deal With Them? , 2006, CLEF.

[123]  Paolo Rosso,et al.  Counting Co-occurrences in Citations to Identify Plagiarised Text Fragments , 2013, CLEF.

[124]  Valentin Jijkoun,et al.  Overview of WebCLEF 2007 , 2008, CLEF.

[125]  Giuseppe Santucci,et al.  Cumulated Relative Position: A Metric for Ranking Evaluation , 2012, IIR.

[126]  Tao Xu,et al.  The Importance of Visual Context Clues in Multimedia Translation , 2011, CLEF.

[127]  Solange de L. Pertile,et al.  A Test Collection to Evaluate Plagiarism by Missing or Incorrect References , 2012, CLEF.

[128]  Se-Jong Kim,et al.  Subtopic Mining Based on Head-Modifier Relation and Co-occurrence of Intents Using Web Documents , 2013, CLEF.

[129]  John Tait,et al.  CLEF-IP 2009: Retrieval Experiments in the Intellectual Property Domain , 2009, CLEF.

[130]  Valentin Jijkoun,et al.  Overview of WebCLEF 2008 , 2008, CLEF.

[131]  Sanda M. Harabagiu,et al.  The Impact of Belief Values on the Identification of Patient Cohorts , 2013, CLEF.

[132]  Allan Hanbury,et al.  Overview of the ImageCLEFphoto 2007 Photographic Retrieval Task , 2008, CLEF.

[133]  Nozha Boujemaa,et al.  The ImageCLEF 2012 Plant Identification Task , 2012, CLEF.

[134]  Fabian Steeg,et al.  Information-Retrieval: Evaluation , 2010 .

[135]  Paul D. Clough,et al.  Selecting Success Criteria: Experiences with an Academic Library Catalogue , 2013, CLEF.

[136]  Maria Gäde,et al.  Which Log for Which Information? Gathering Multilingual Data from Different Log File Types , 2010, CLEF.

[137]  Rada Mihalcea,et al.  Improving Query Expansion for Image Retrieval via Saliency and Picturability , 2011, CLEF.

[138]  Patrick Juola,et al.  An Overview of the Traditional Authorship Attribution Subtask , 2012, CLEF.

[139]  Valentin Jijkoun,et al.  Overview of the CLEF 2006 Multilingual Question Answering Track , 2006, CLEF.

[140]  Stefanie Nowak,et al.  The CLEF 2011 Photo Annotation and Concept-based Retrieval Tasks , 2011, CLEF.

[141]  Preben Hansen,et al.  Going beyond CLEF-IP: The 'Reality' for Patent Searchers? , 2012, CLEF.

[142]  Guanglai Gao,et al.  Hidden Markov Model for Term Weighting in Verbose Queries , 2012, CLEF.

[143]  Qiang Zhang,et al.  A Web-Based CLIR System with Cross-Lingual Topical Pseudo Relevance Feedback , 2013, CLEF.

[144]  Gabriella Kazai,et al.  Overview of INEX 2013 , 2013, CLEF.

[145]  Adrian Popescu,et al.  Overview of the Wikipedia Retrieval Task at ImageCLEF 2010 , 2010, CLEF.

[146]  Nicola Ferro,et al.  Cultural Heritage in CLEF (CHiC) Overview 2012 , 2012, CLEF.

[147]  Maximilian Eibl,et al.  Comparing IR System Components Using Beanplots , 2012, CLEF.

[148]  Mohand Boughanem,et al.  Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation , 2010, CLEF.

[149]  Allan Hanbury,et al.  Bringing the Algorithms to the Data: Cloud-Based Benchmarking for Medical Image Analysis , 2012, CLEF.

[150]  Carol Peters,et al.  Cross-Language Information Retrieval (CLIR) Track Overview , 1997, TREC.

[151]  Axel-Cyrille Ngonga Ngomo,et al.  Overview of CLEF Question Answering Track 2014 , 2014, CLEF.

[152]  Eugene Kim,et al.  Overview of the ImageCLEFmed 2006 Medical Retrieval and Medical Annotation Tasks , 2006, CLEF.

[153]  Gabriella Kazai,et al.  Overview of INEX 2014 , 2014, CLEF.

[154]  Thomas Martin Deserno,et al.  Medical Image Annotation in ImageCLEF 2008 , 2008, CLEF.

[155]  Ville T. Turunen,et al.  Advances in Multilingual and MultiModal Information Retrieval, 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers , 2009 .

[156]  Craig MacDonald,et al.  Comparing Algorithms for Microblog Summarisation , 2014, CLEF.

[157]  Philipp Cimiano,et al.  Overview of the Cross-lingual Expert Search (CriES) Pilot Challenge , 2010, CLEF.

[158]  Stefanie Nowak,et al.  New Strategies for Image Annotation: Overview of the Photo Annotation Task at ImageCLEF 2010 , 2010, CLEF.

[159]  Barbara Caputo,et al.  The Robot Vision Task , 2010, ImageCLEF.

[160]  Fabio Crestani,et al.  Overview of the International Sexual Predator Identification Competition at PAN-2012 , 2012, CLEF.

[161]  Julio Gonzalo,et al.  Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems , 2013, CLEF.

[162]  Shlomo Geva,et al.  Overview of the INEX 2012 Relevance Feedback Track , 2012, CLEF.

[163]  Diana Santos,et al.  GikiCLEF: Expectations and Lessons Learned , 2009, CLEF.

[164]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[165]  Renata de Matos Galante,et al.  A New Approach for Cross-Language Plagiarism Analysis , 2010, CLEF.

[166]  Jaap Kamps,et al.  Overview of the INEX 2013 Linked Data Track , 2013, CLEF.

[167]  Martin Braschler,et al.  PROMISE retreat report prospects and opportunities for information access evaluation , 2012, SIGF.

[168]  Kalervo Järvelin,et al.  A Dictionary- and Corpus-Independent Statistical Lemmatizer for Information Retrieval in Low Resource Languages , 2010, CLEF.

[169]  Azadeh Shakery,et al.  Authorship Identification Using Dynamic Selection of Features from Probabilistic Feature Set , 2014, CLEF.

[170]  Azadeh Shakery,et al.  Exploiting Multiple Translation Resources for English-Persian Cross Language Information Retrieval , 2013, CLEF.

[171]  Martin Braschler,et al.  CLEF 2001 - Overview of Results , 2001, CLEF.

[172]  Pavel Braslavski,et al.  Towards Automatic Evaluation of Health-Related CQA Data , 2014, CLEF.

[173]  Mark Sanderson,et al.  The CLEF 2004 Cross-Language Image Retrieval Track , 2004, CLEF.

[174]  Julio Gonzalo,et al.  WePS3 Evaluation Campaign: Overview of the On-line Reputation Management Task , 2010, CLEF.

[175]  Carol Peters,et al.  CLEF 2009 Ad Hoc Track Overview: TEL and Persian Tasks , 2009, CLEF.

[176]  Elaine Toms,et al.  Cultural Heritage in CLEF (CHiC) 2013 , 2013, CLEF.

[177]  Maximilian Eibl,et al.  Improving Transcript-Based Video Retrieval Using Unsupervised Language Model Adaptation , 2014, CLEF.

[178]  Carol Peters,et al.  Analysis combination and pseudo relevance feedback in conceptual language model , 2010 .

[179]  Fredric C. Gey,et al.  GeoCLEF 2008: the CLEF 2008 Cross-Language Geographic Information Retrieval Track Overview , 2008, Conference and Labs of the Evaluation Forum.

[180]  Nicola Ferro,et al.  CHiC 2011 - Cultural Heritage in CLEF: From Use Cases to Evaluation in Practice for Multilingual Information Access to Cultural Heritage , 2011, CLEF.

[181]  M. Felisa Verdejo,et al.  Overview of the Answer Validation Exercise 2007 , 2007, CLEF.

[182]  Gareth J. F. Jones,et al.  The CLEF 2003 Cross-Language Spoken Document Retrieval Track , 2003, CLEF.

[183]  Carol Peters.  Cross-Language Evaluation Forum - CLEF 2008 , 2008 .

[184]  Henning Müller,et al.  Overview of the ImageCLEF 2012 Medical Image Retrieval and Classification Tasks , 2012, CLEF.

[185]  Maria Kvist,et al.  SCAN: A Swedish Clinical Abbreviation Normalizer - Further Development and Adaptation to Radiology , 2014, CLEF.

[186]  Theodora Tsikrika,et al.  Overview of the WikipediaMM Task at ImageCLEF 2009 , 2009, CLEF.

[187]  Sanna Salanterä,et al.  Overview of the ShARe/CLEF eHealth Evaluation Lab 2013 , 2013, CLEF.

[188]  Nicola Orio,et al.  MusiClef: Multimodal Music Tagging Task , 2012, CLEF.

[189]  Craig MacDonald,et al.  A Study of Personalised Medical Literature Search , 2014, CLEF.

[190]  David Zellhöfer.  Overview of the Personal Photo Retrieval Pilot Task at ImageCLEF 2012 , 2012, CLEF.

[191]  Yasser El-Sonbaty,et al.  ALQASIM: Arabic Language Question Answer Selection in Machines , 2013, CLEF.

[192]  Jörg Tiedemann,et al.  CLEF 2007 Working Notes , 2007 .

[193]  M. de Rijke,et al.  Overview of WebCLEF 2005 , 2005, CLEF.

[194]  Anselmo Peñas,et al.  Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation , 2009, CLEF.

[195]  M. de Rijke,et al.  On the Evaluation of Entity Profiles , 2010, CLEF.

[196]  Mark Stevenson,et al.  Self-supervised Relation Extraction Using UMLS , 2014, CLEF.

[197]  Michael Kluck,et al.  Domain-Specific Track CLEF 2006: Overview of Results and Approaches, Remarks on the Assessment Analysis , 2006, CLEF.

[198]  Frank Hopfgartner,et al.  Benchmarking News Recommendations in a Living Lab , 2014, CLEF.

[199]  Julio Gonzalo,et al.  The CLEF 2003 Interactive Track , 2003, CLEF.

[200]  Levente Kovács,et al.  The Appearance of the Giant Component in Descriptor Graphs and Its Application for Descriptor Selection , 2012, CLEF.

[201]  Giuseppe Santucci,et al.  To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma? , 2011, CLEF.

[202]  Barbara Caputo,et al.  Overview of the CLEF 2009 Medical Image Annotation Track , 2009, CLEF.

[203]  Gareth J. F. Jones,et al.  CLEF 2002 Cross-Language Spoken Document Retrieval Pilot Track Report , 2002, CLEF.

[204]  Carol Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems , 2003, Lecture Notes in Computer Science.

[205]  Alan F. Smeaton,et al.  The scholarly impact of TRECVid (2003-2009) , 2011, J. Assoc. Inf. Sci. Technol..

[206]  Gabriella Kazai,et al.  Overview of the INEX 2014 Social Book Search Track , 2014, CLEF.

[207]  Krisztian Balog,et al.  Towards a Living Lab for Information Retrieval Research and Development - A Proposal for a Living Lab for Product Search Tasks , 2011, CLEF.

[208]  Adrian Popescu,et al.  Overview of the Wikipedia Image Retrieval Task at ImageCLEF 2011 , 2011, CLEF.

[209]  Julio Gonzalo,et al.  Advances in Cross-Language Information Retrieval , 2002, Lecture Notes in Computer Science.

[210]  María-Dolores Olvera-Lobo,et al.  Multilingual Question-Answering System in Biomedical Domain on the Web: An Evaluation , 2011, CLEF.

[211]  Georgios Paliouras,et al.  The Effect of Dimensionality Reduction on Large Scale Hierarchical Classification , 2014, CLEF.

[212]  Matthias Hagen,et al.  Supporting More-Like-This Information Needs: Finding Similar Web Content in Different Scenarios , 2014, CLEF.

[213]  Eduard H. Hovy,et al.  Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation , 2011, CLEF.

[214]  Thomas Deselaers,et al.  The Visual Concept Detection Task in ImageCLEF 2008 , 2008, CLEF.

[215]  Fredric C. Gey,et al.  GeoCLEF: the CLEF 2005 Cross-Language Geographic Information Retrieval Track , 2005, CLEF.

[216]  Giuseppe Santucci,et al.  Improving Ranking Evaluation Employing Visual Analytics , 2013, CLEF.

[217]  Ralf Bierig,et al.  A Hybrid Approach for Multi-faceted IR in Multimodal Domain , 2014, CLEF.

[218]  Maarten Marx,et al.  Evaluation Methods for Rankings of Facetvalues for Faceted Search , 2011, CLEF.

[219]  Mikko Kurimo,et al.  Overview and Results of Morpho Challenge 2009 , 2009, CLEF.

[220]  Luis A. Leiva,et al.  Relevant Clouds: Leveraging Relevance Feedback to Build Tag Clouds for Image Search , 2013, CLEF.

[221]  Mihai Lupu,et al.  A Real-World Framework for Translator as Expert Retrieval , 2014, CLEF.

[222]  Julio Gonzalo,et al.  iCLEF 2004 Track Overview: Pilot Experiments in Interactive Cross-Language Question Answering , 2004, CLEF.

[223]  Maximilian Eibl,et al.  A Quantitative Look at the CLEF Working Notes , 2013, CLEF.

[224]  Pavel Pecina,et al.  Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval , 2012, CLEF.

[225]  Allan Hanbury,et al.  Effects of Language and Topic Size in Patent IR: An Empirical Study , 2012, CLEF.

[226]  Dietrich Rebholz-Schuhmann,et al.  Context-Dependent Semantic Annotation in Cross-Lingual Biomedical Resources , 2013, CLEF.

[227]  Eneko Agirre,et al.  Advances in Multilingual and Multimodal Information Retrieval. , 2008 .

[228]  Vasudeva Varma,et al.  A Language-Independent Approach to Identify the Named Entities in Under-Resourced Languages and Clustering Multilingual Documents , 2011, CLEF.

[229]  Carol Peters,et al.  CLEF 2005: Ad Hoc Track Overview , 2005, CLEF.

[230]  Philipp Schaer,et al.  Better than Their Reputation? On the Reliability of Relevance Assessments with Students , 2012, CLEF.

[231]  Julio Gonzalo,et al.  WePS-3 Evaluation Campaign: Overview of the Web People Search Clustering and Attribute Extraction Tasks , 2010, CLEF.

[232]  Azadeh Shakery,et al.  Creating a Persian-English Comparable Corpus , 2010, CLEF.

[233]  Ryen W. White,et al.  Overview of the CLEF-2006 Cross-Language Speech Retrieval Track , 2006, CLEF.

[234]  M. de Rijke,et al.  Generating Pseudo Test Collections for Learning to Rank Scientific Articles , 2012, CLEF.

[235]  Nozha Boujemaa,et al.  The CLEF 2011 Plant Images Classification Task , 2011, CLEF.

[236]  Benno Stein,et al.  CLEF 2013 , 2013, SIGIR Forum.

[237]  Valentin Jijkoun,et al.  Overview of the CLEF 2007 Multilingual Question Answering Track , 2007, CLEF.

[238]  Elaine Toms,et al.  Building a Common Framework for IIR Evaluation , 2013, CLEF.

[239]  Nicola Ferro,et al.  CLEF 15th Birthday: What Can We Learn From Ad Hoc Retrieval? , 2014, CLEF.

[240]  Erhard Rahm,et al.  The Scholarly Impact of CLEF (2000-2009) , 2013, CLEF.

[241]  Alistair Moffat,et al.  Improvements that don't add up: ad-hoc retrieval results since 1998 , 2009, CIKM.

[242]  Stefan Baerisch,et al.  The Domain-Specific Track at CLEF 2006: Overview of Approaches, Results and Assessment , 2006, CLEF.

[243]  Bogdan Sacaleanu,et al.  Overview of the CLEF 2008 Multilingual Question Answering Track , 2008, CLEF.

[244]  Marina Bosch,et al.  ImageCLEF, Experimental Evaluation in Visual Information Retrieval , 2010 .

[245]  Jesus Martínez-Gómez,et al.  Overview of the ImageCLEF 2013 Robot Vision Task , 2012, CLEF.

[246]  Carol Peters,et al.  The Future of Large-Scale Evaluation Campaigns for Information Retrieval in Europe , 2007, ECDL.

[247]  Michael Kluck.  The Domain-Specific Track in CLEF 2004: Overview of the Results and Remarks on the Assessment Process , 2004, CLEF.

[248]  Miguel Cazorla,et al.  ImageCLEF 2013: The Vision, the Data and the Open Challenges , 2013, CLEF.

[249]  Paul Clough,et al.  Information Access Evaluation. Multilinguality, Multimodality, and Interaction , 2014, Lecture Notes in Computer Science.

[250]  Julio Gonzalo,et al.  Overview of the CLEF 2005 Interactive Track , 2005, CLEF.

[251]  Carol Peters,et al.  Cross-Language Evaluation Forum: Objectives, Results, Achievements , 2004, Information Retrieval.

[252]  Pavel Braslavski,et al.  Search Snippet Evaluation at Yandex: Lessons Learned and Future Directions , 2011, CLEF.

[253]  Nozha Boujemaa,et al.  The imageCLEF plant identification task 2013 , 2013, MAED '13.

[254]  Hervé Glotin,et al.  LifeCLEF 2014: Multimedia Life Species Identification Challenges , 2014, CLEF.

[255]  James Allan,et al.  Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne , 2012, SIGF.