One Search Engine or Two for Question-Answering

We present here a preliminary analysis of the results of our runs in the Question Answering track of TREC9. We have developed a complete system, including our own indexer and search engine, GuruQA, which provides document result lists that our Answer Selection module processes to identify answer fragments. Some TREC participants use a standard set of result lists provided by AT&T’s running of the SMART search engine. We wondered how our results would be affected by using the AT&T result sets. For a variety of reasons we could not replace GuruQA’s results with SMART’s, but we could use document co-occurrence counts to influence our hit-lists. We submitted two runs to NIST for both the 50- and 250-byte cases, one with and one without consideration of the AT&T document result sets. The AT&T document set was only used for a subset of about a third of the questions. This subset exhibited an increase in Mean Reciprocal Answer Rank score of 13% and 8% for the two tasks.

[1]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[2]  George A. Miller.  WordNet: A Lexical Database for English , 1992, HLT.

[3]  W. Bruce Croft,et al.  Evaluation of an inference network-based retrieval model , 1991, TOIS.

[4]  Yael Ravin,et al.  Identifying and extracting relations from text , 1999 .

[5]  Dragomir R. Radev,et al.  Ranking suspected answers to natural language questions using predictive annotation , 2000, ANLP.

[6]  Nina Wacholder,et al.  Disambiguation of Proper Names in Text , 1997, ANLP.

[7]  Dragomir R. Radev,et al.  The Use of Predictive Annotation for Question Answering in TREC8 , 1999, TREC.

[8]  Paul B. Kantor,et al.  A study of information seeking and retrieving. III. Searchers, searches, and overlap , 1988, J. Am. Soc. Inf. Sci..

[9]  Eric W. Brown,et al.  The GURU System in TREC-6 , 1997, TREC.

[10]  Eric Brill,et al.  Classifier Combination for Improved Lexical Disambiguation , 1998, ACL.

[11]  Chris Buckley,et al.  Implementation of the SMART Information Retrieval System , 1985 .

[12]  Rada Mihalcea,et al.  Using WordNet and Lexical Operators to Improve Internet Searches , 2000, IEEE Internet Comput..

[13]  Nicholas J. Belkin,et al.  The effect multiple query representations on information retrieval system performance , 1993, SIGIR.

[14]  I MoldovanDan,et al.  Using WordNet and Lexical Operators to Improve Internet Searches , 2000 .

[15]  Susan T. Dumais,et al.  Personalized information delivery: an analysis of information filtering methods , 1992, CACM.

[16]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[17]  Wayne D. Gray,et al.  Basic objects in natural categories , 1976, Cognitive Psychology.

[18]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[19]  Dragomir R. Radev,et al.  Question-answering by predictive annotation , 2000, SIGIR '00.