Question answering at the cross-language evaluation forum 2003–2010

The paper offers an overview of the key issues raised during the 8 years’ activity of the Multilingual Question Answering Track at the Cross Language Evaluation Forum (CLEF). The general aim of the track has been to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages, also drawing attention to a number of challenging issues for research in multilingual QA. The paper gives a brief description of how the task has evolved over the years and of the way in which the data sets have been created, presenting also a short summary of the different types of questions developed. The document collections adopted in the competitions are outlined as well, and data about participation is provided. Moreover, the main measures used to evaluate system performances are explained and an overall analysis of the results achieved is presented.

[1]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track Evaluation , 2000, TREC.

[2]  Ellen M. Voorhees,et al.  The seventh text REtrieval conference (TREC-7) , 1999 .

[3]  Ellen M. Voorhees,et al.  Overview of the TREC-9 Question Answering Track , 2000, TREC.

[4]  Dragomir R. Radev,et al.  Question-answering by predictive annotation , 2000, SIGIR '00.

[5]  Charles L. A. Clarke,et al.  Exploiting redundancy in question answering , 2001, SIGIR '01.

[6]  Lynette Hirschman,et al.  Natural language question answering: the view from here , 2001, Natural Language Engineering.

[7]  Eduard H. Hovy,et al.  Toward Semantics-Based Answer Pinpointing , 2001, HLT.

[8]  Andrew G. Clark,et al.  Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) , 2002 .

[9]  Bernardo Magnini,et al.  Is It the Right Answer? Exploiting Web Redundancy for Answer Validation , 2002, ACL.

[10]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[11]  Gilad Mishne,et al.  The University of Amsterdam at QA@CLEF 2004 , 2003, CLEF.

[12]  Maarten de Rijke,et al.  The Multiple Language Question Answering Track at CLEF 2003 , 2003, CLEF.

[13]  Bernardo Magnini,et al.  Bridging Languages for Question Answering: DIOGENE at CLEF 2003 , 2003, CLEF.

[14]  Fernando Llopis,et al.  Question Answering in Spanish , 2003, CLEF.

[15]  Sven Hartrumpf.  Question Answering using Sentence Parsing and Semantic Network Matching , 2004, CLEF.

[16]  Maarten de Rijke,et al.  Overview of the CLEF 2004 Multilingual Question Answering Track , 2004, CLEF.

[17]  Fernando Llopis,et al.  Does English help Question Answering in Spanish? , 2004, CLEF.

[18]  M. Felisa Verdejo,et al.  Question Answering Pilot Task at CLEF 2004 , 2004, CLEF.

[19]  M. de Rijke,et al.  Overview of the CLEF 2005 Multilingual Question Answering Track , 2005, CLEF.

[20]  Dominique Laurent,et al.  Cross Lingual Question Answering using QRISTAL for CLEF 2008 , 2006, CLEF.

[21]  Jörg Tiedemann,et al.  Question Answering for Dutch using Dependency Relations , 2005, CLEF.

[22]  André F. T. Martins,et al.  Priberam's Question Answering System for Portuguese , 2005, CLEF.

[23]  C. Peters,et al.  Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science) , 2005 .

[24]  CLEF 2005 , Wien , 21-23 september 2005 Cross Lingual Question Answering using QRISTAL for CLEF 2005 , 2005 .

[25]  Julio Gonzalo,et al.  Overview of the CLEF 2005 Interactive Track , 2005, CLEF.

[26]  Adán Cassan,et al.  Priberam's Question Answering System in a Cross-Language Environment , 2006, CLEF.

[27]  Laurianne Sitbon,et al.  The LIA at QA@CLEF-2006 , 2006, CLEF.

[28]  M. Felisa Verdejo,et al.  Overview of the Answer Validation Exercise 2007 , 2006, CLEF.

[29]  Valentin Jijkoun,et al.  Overview of the WiQA Task at CLEF 2006 , 2006, CLEF.

[30]  Ellen M. Voorhees,et al.  Overview of the TREC 2006 , 2007, TREC.

[31]  Fredric C. Gey,et al.  Accessing Multilingual Information Repositories (6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005) , 2006 .

[32]  Jimmy J. Lin,et al.  Overview of the TREC 2006 ciQA task , 2007, SIGF.

[33]  John Prager.  Open-Domain Question Answering: Foundations and Trends(R) in Information Retrieval , 2007 .

[34]  Dan I. Moldovan,et al.  Multilingual Question Answering through Intermediate Translation: LCC's PowerAnswer at QA@CLEF 2007 , 2007, CLEF.

[35]  Adán Cassan,et al.  Priberam's Question Answering System in QA@CLEF 2007 , 2007, CLEF.

[36]  Fernando Llopis,et al.  Evaluation of Open-Domain Question Answering Systems within a Time Constraint , 2007, 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07).

[37]  Constantin Orasan,et al.  University of Wolverhampton at CLEF 2007 , 2008, CLEF.

[38]  Dan Tufis,et al.  RACAI's Question Answering System at QA@CLEF 2007 , 2007, CLEF.

[39]  M. Felisa Verdejo,et al.  Overview of the Answer Validation Exercise 2007 , 2007, CLEF.

[40]  Paulo Quaresma,et al.  The Senso Question Answering Approach to Portuguese QA@CLEF-2007 , 2007, CLEF.

[41]  Valentin Jijkoun,et al.  Overview of the CLEF 2007 Multilingual Question Answering Track , 2007, CLEF.

[42]  C. J. van Rijsbergen,et al.  Information Retrieval , 1979, Encyclopedia of GIS.

[43]  Adán Cassan,et al.  Priberam's Question Answering System in QA@CLEF 2007 , 2007, CLEF.

[44]  Paulo Quaresma,et al.  The Senso Question Answering System at QA@CLEF 2008 , 2008, CLEF.

[45]  Bogdan Sacaleanu,et al.  Overview of the CLEF 2008 Multilingual Question Answering Track , 2008, CLEF.

[46]  Lori Lamel,et al.  Question Answering on Speech Transcriptions: the QAST evaluation in CLEF , 2008, LREC.

[47]  Adrian Iftene,et al.  Question Answering on English and Romanian Languages , 2009, CLEF.

[48]  Lourdes Araujo,et al.  Information Retrieval Baselines for the ResPubliQA Task , 2009, CLEF.

[49]  Enrico Motta,et al.  Cross ontology query answering on the semantic web: an initial evaluation , 2009, K-CAP '09.

[50]  Anselmo Peñas,et al.  Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation , 2009, CLEF.

[51]  Vikash Kumar,et al.  Towards Interactive Question Answering: An Ontology-Based Approach , 2009, 2009 IEEE International Conference on Semantic Computing.

[52]  Bonnie L. Webber,et al.  Special issue on interactive question answering: Introduction , 2009, Natural Language Engineering.

[53]  Lourdes Araujo,et al.  Approaching Question Answering by means of Paragraph Validation , 2009, CLEF.

[54]  Verginica Barbu Mititelu,et al.  A Trainable Multi-factored QA System , 2009, CLEF.

[55]  Paolo Rosso,et al.  NLEL-MAAT at CLEF-ResPubliQA , 2009, CLEF.

[56]  Diana Santos,et al.  GikiCLEF: Crosscultural Issues in an International Setting: Asking non-English-centered Questions to Wikipedia , 2009, CLEF.

[57]  Jennifer Chu-Carroll,et al.  Building Watson: An Overview of the DeepQA Project , 2010, AI Mag..

[58]  Arantxa Otegi,et al.  Document Expansion for Cross-Lingual Passage Retrieval , 2010, CLEF.

[59]  Ruli Manurung,et al.  Contextual Approach for Paragraph Selection in Question Answering Task , 2010, CLEF.

[60]  T. Maung.  on in C , 2010 .

[61]  Diego Mollá Aliod,et al.  Question Answering , 2010, Handbook of Natural Language Processing.

[62]  Anselmo Peñas,et al.  A Simple Measure to Assess Non-response , 2011, ACL.

[63]  Günter Neumann,et al.  The QALL-ME Framework: A specifiable-domain multilingual Question Answering architecture , 2011, J. Web Semant..