How Frogs Built the Berlin Wall: A Detailed Error Analysis of a Question Answering System for Dutch

The paper describes the University of Amsterdam’s participation in the Question Answering track at CLEF 2003, our system and the results produced by it. A thorough analysis of the wrong answers given by our system is provided, including a discussion of each type of error and possible strategies for handling them. We outline our current efforts for improvement of the system, and propose additional research directions and procedures to reduce errors of the presented types.

[1]  Scott Miller,et al.  TREC 2002 QA at BBN: Answer Selection and Confidence Estimation , 2002, TREC.

[2]  Nelleke Oostdijk,et al.  The Spoken Dutch Corpus. Overview and First Evaluation , 2000, LREC.

[3]  Nelleke Oostdijk,et al.  The spoken Dutch Corpus. Outline and first evaluation , 2000 .

[4]  Jimmy J. Lin,et al.  Data-Intensive Question Answering , 2001, TREC.

[5]  Bernardo Magnini,et al.  Is It the Right Answer? Exploiting Web Redundancy for Answer Validation , 2002, ACL.

[6]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[7]  Jimmy J. Lin,et al.  Extracting Answers from the Web Using Data Annotation and Knowledge Mining Techniques , 2002, TREC.

[8]  Ellen M. Voorhees,et al.  The Tenth Text REtrieval Conference, TREC 2001 | NIST , 2002 .

[9]  Gilad Mishne,et al.  Preprocessing documents to answer Dutch questions , 2003 .

[10]  Jennifer Chu-Carroll,et al.  A Multi-Strategy and Multi-Source Approach to Question Answering , 2002, TREC.

[11]  Eduard H. Hovy,et al.  Learning surface text patterns for a Question Answering System , 2002, ACL.

[12]  Daniel Marcu,et al.  A Noisy-Channel Approach to Question Answering , 2003, ACL.

[13]  Jimmy J. Lin,et al.  AskMSR: Question Answering Using the Worldwide Web , 2002 .

[14]  Sanda M. Harabagiu,et al.  LCC Tools for Question Answering , 2002, TREC.

[15]  Dell Zhang,et al.  Question classification using support vector machines , 2003, SIGIR.

[16]  France T́elécom.  Learning Paraphrases to Improve a Question-Answering System , 2003 .

[17]  Gilad Mishne,et al.  The University of Amsterdam at QA@CLEF 2004 , 2003, CLEF.

[18]  M. de Rijke,et al.  Tequesta: The University of Amsterdam's Textual Question Answering System , 2001, TREC.

[19]  Salim Roukos,et al.  IBM's Statistical Question Answering System-TREC 11 , 2001, TREC.

[20]  Thorsten Brants,et al.  TnT – A Statistical Part-of-Speech Tagger , 2000, ANLP.