Quantitative evaluation of passage retrieval algorithms for question answering

Passage retrieval is an important component common to many question answering systems. Because most evaluations of question answering systems focus on end-to-end performance, comparison of common components becomes difficult. To address this shortcoming, we present a quantitative evaluation of various passage retrieval algorithms for question answering, implemented in a framework called Pauchok. We present three important findings: Boolean querying schemes perform well in the question answering task. The performance differences between various passage retrieval algorithms vary with the choice of document retriever, which suggests significant interactions between document retrieval and passage retrieval. The best algorithms in our evaluation employ density-based measures for scoring query terms. Our results reveal future directions for passage retrieval and question answering.

[1]  William S. Cooper,et al.  Getting beyond Boole , 1988, Inf. Process. Manag..

[2]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[3]  James Allan,et al.  Approaches to passage retrieval in full text information systems , 1993, SIGIR.

[4]  James P. Callan,et al.  Passage-level evidence in document retrieval , 1994, SIGIR '94.

[5]  Stephen E. Robertson,et al.  GatfordCentre for Interactive Systems ResearchDepartment of Information , 1996 .

[6]  Stephen E. Robertson,et al.  Okapi at TREC-4 , 1995, TREC.

[7]  Wei Li,et al.  Information Extraction Supported Question Answering , 1999, TREC.

[8]  Ellen M. Voorhees,et al.  Overview of the TREC-9 Question Answering Track , 2000, TREC.

[9]  Sanda M. Harabagiu,et al.  Experiments with Open-Domain Textual Question Answering , 2000, COLING.

[10]  Charles L. A. Clarke,et al.  Relevance ranking for one to three term queries , 1997, Inf. Process. Manag..

[11]  Charles L. A. Clarke,et al.  Question Answering by Passage Selection (MultiText Experiments for TREC-9) , 2000, TREC.

[12]  Adwait Ratnaparkhi,et al.  IBM's Statistical Question Answering System , 2000, TREC.

[13]  Eduard H. Hovy,et al.  The Use of External Knowledge of Factoid QA , 2001, TREC.

[14]  Lynette Hirschman,et al.  Natural language question answering: the view from here , 2001, Natural Language Engineering.

[15]  Martin M. Soubbotin.  Patterns of Potential Answer Expressions as Clues to the Right Answers , 2001, TREC.

[16]  Sanda M. Harabagiu,et al.  Answering Complex, List and Context Questions with LCC's Question-Answering Server , 2001, TREC.

[17]  Fernando Llopis,et al.  University of Alicante at TREC-10 , 2001, TREC.

[18]  Salim Roukos,et al.  IBM's Statistical Question Answering System-TREC 11 , 2001, TREC.

[19]  Jungyun Seo,et al.  SiteQ: Engineering High Performance QA System Using Lexico-Semantic Pattern Matching and Shallow NLP , 2001, TREC.

[20]  Fernando Llopis,et al.  IR-n: A Passage Retrieval System at CLEF-2001 , 2001, CLEF.

[21]  Gideon S. Mann,et al.  Analyses for elucidating current question answering technology , 2001, Natural Language Engineering.

[22]  Ellen M. Voorhees,et al.  Overview of the TREC 2002 Question Answering Track , 2003, TREC.

[23]  Performance Issues and Error Analysis in an Open-Domain Question Answering System , 2002, ACL.

[24]  Jimmy J. Lin,et al.  The role of context in question answering systems , 2003, CHI Extended Abstracts.

[25]  Jimmy J. Lin,et al.  What Makes a Good Answer? The Role of Context in Question Answering , 2003, INTERACT.

[26]  Jimmy J. Lin,et al.  Selectively Using Relations to Improve Precision in Question Answering , 2003 .