From document retrieval to question answering

is one of the most valuable goods in modern society. With the rise of computers, storing huge amounts of data has become efficient and inexpensive. Although we are now in a position where we have unprecedented amounts of information at our finger tips, the question arises how to access these large amounts of data to find the information one is interested in. The issue of developing methods and tools for finding automatically relevant information is addressed by the research area of information retrieval, and, over the last decades, sophisticated document retrieval systems have been developed. One particular branch of information retrieval is question answering. Question answering systems enable users to pose full natural language questions, as opposed to keyword-based queries, which are commonly used in document retrieval. In recent years, question answering has witnessed a renaissance, which is mainly due to the availability of large corpora. Current question answering systems depend strongly on document retrieval as a means for identifying documents that are likely to contain answer to a given question. This thesis investigates the usefulness of different standard and novel document retrieval approaches in the context of question answering. More specifically, it compares them with respect to their ability to identify documents containing a correct answer. In addition, we also investigate to what extent the quality of a particular document retrieval approach has an impact on the overall performance of a specific question answering system.

[1]  Justin Zobel,et al.  Term-ordered query evaluation versus document-ordered query evaluation for large document databases , 1998, SIGIR '98.

[2]  Hans Peter Luhn,et al.  The Automatic Creation of Literature Abstracts , 1958, IBM J. Res. Dev..

[3]  G. A. Barnard,et al.  Transmission of Information: A Statistical Theory of Communications. , 1961 .

[4]  Marti A. Hearst,et al.  Adaptive Sentence Boundary Disambiguation , 1994, ANLP.

[5]  Oren Etzioni,et al.  Scaling question answering to the Web , 2001, WWW '01.

[6]  David Hawking,et al.  Overview of the TREC-2002 Web Track , 2002, TREC.

[7]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[8]  Hsinchun Chen,et al.  A Machine Learning Approach to Inductive Query by Examples: An Experiment Using Relevance Feedback, ID3, Genetic Algorithms, and Simulated Annealing , 1998, J. Am. Soc. Inf. Sci..

[9]  Ellen M. Voorhees,et al.  The Sixth Text REtrieval Conference (TREC-6) , 2000, Inf. Process. Manag..

[10]  Dan Roth,et al.  Learning Question Classifiers , 2002, COLING.

[11]  S.J.J. Smith,et al.  Empirical Methods for Artificial Intelligence , 1995 .

[12]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[13]  Inderjeet Mani,et al.  How to Evaluate Your Question Answering System Every Day ... and Still Get Real Work Done , 2000, LREC.

[14]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[15]  Sanda Harabagiu,et al.  High-performance, open-domain question answering from large text collections , 2001 .

[16]  Marcel Worring,et al.  NIST Special Publication , 2005 .

[17]  Ellen M. Voorhees,et al.  Evaluation by highly relevant documents , 2001, SIGIR '01.

[18]  Ellen M. Voorhees,et al.  Overview of the Seventh Text REtrieval Conference , 1998 .

[19]  L J Kitchen,et al.  Exploring Statistics: A Modern Introduction to Data Analysis and Inference , 1987 .

[20]  Ellen M. Voorhees,et al.  Evaluating evaluation measure stability , 2000, SIGIR '00.

[21]  Stephen E. Robertson,et al.  Okapi/Keenbow at TREC-8 , 1999, TREC.

[22]  Jimmy J. Lin,et al.  What Makes a Good Answer? The Role of Context in Question Answering , 2003, INTERACT.

[23]  Julian Kupiec,et al.  MURAX: a robust linguistic approach for question answering using an on-line encyclopedia , 1993, SIGIR.

[24]  Harold Borko,et al.  Automatic indexing , 1981, ACM '81.

[25]  Bert F. Green,et al.  Baseball: an automatic question-answerer , 1899, IRE-AIEE-ACM '61 (Western).

[26]  A. V. Phillips,et al.  A Question-Answering Routine , 1960 .

[27]  David Hawking,et al.  Relevance weighting using distance between term occurrences , 1996 .

[28]  Ron Kohavi,et al.  Error-Based and Entropy-Based Discretization of Continuous Features , 1996, KDD.

[29]  Alistair Moffat,et al.  Locality-Based Information Retrieval , 1999, Australasian Database Conference.

[30]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[31]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[32]  Alan F. Smeaton,et al.  The effect of pool depth on system evaluation in TREC , 2001, J. Assoc. Inf. Sci. Technol..

[33]  M VoorheesEllen The TREC question answering track , 2001 .

[34]  David Hawking,et al.  Proximity Operators - So Near And Yet So Far , 1995, TREC.

[35]  Mark Sanderson,et al.  University of Sheffield TREC-8 Q&A System , 1999, TREC.

[36]  Irene A. G. Roberts,et al.  Information Retrieval for Question Answering , 2003 .

[37]  Bernardo Magnini,et al.  Is It the Right Answer? Exploiting Web Redundancy for Answer Validation , 2002, ACL.

[38]  Gideon S. Mann Fine-Grained Proper Noun Ontologies for Question Answering , 2002, COLING 2002.

[39]  Jimmy J. Lin,et al.  Web question answering: is more always better? , 2002, SIGIR '02.

[40]  Antonietta Alonge,et al.  ItalWordNet: a Large Semantic Database for Italian , 2000, LREC.

[41]  Donna K. Harman,et al.  Ranking Algorithms , 1992, Information Retrieval: Data Structures & Algorithms.

[42]  Martin M. Soubbotin,et al.  Use of Patterns for Detection of Likely Answer Strings: A Systematic Approach , 2002, TREC.

[43]  Thomas G. Dietterich Machine-Learning Research Four Current Directions , 1997 .

[44]  Alistair Moffat,et al.  Effective document presentation with a locality-based similarity heuristic , 1999, SIGIR '99.

[45]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track Evaluation , 2000, TREC.

[46]  Dekang Lin,et al.  Dependency-Based Evaluation of Minipar , 2003 .

[47]  Geoffrey Sampson,et al.  English for the Computer: The SUSANNE Corpus and Analytic Scheme , 1995, Computational Linguistics.

[48]  Yong Wang,et al.  Using Model Trees for Classification , 1998, Machine Learning.

[49]  Charles L. A. Clarke,et al.  Statistical Selection of Exact Answers (MultiText Experiments for TREC 2002) , 2002, TREC.

[50]  Jimmy J. Lin,et al.  Data-Intensive Question Answering , 2001, TREC.

[51]  Charles L. A. Clarke,et al.  The impact of corpus size on question answering performance , 2002, SIGIR '02.

[52]  Christof Monz,et al.  Document Retrieval in the Context of Question Answering , 2003, ECIR.

[53]  Eduard H. Hovy,et al.  Toward Semantics-Based Answer Pinpointing , 2001, HLT.

[54]  Ron Sacks-Davis,et al.  Similarity Measures for Short Queries , 1995, TREC.

[55]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[56]  Sanda M. Harabagiu,et al.  The Role of Lexico-Semantic Feedback in Open-Domain Textual Question-Answering , 2001, ACL.

[57]  Nuel D. Belnap,et al.  The logic of questions and answers , 1976 .

[58]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[59]  Mark T. Maybury Toward a Question Answering Roadmap , 2003, New Directions in Question Answering.

[60]  E. Michael Keen,et al.  Term position ranking: some new test results , 1992, SIGIR '92.

[61]  Marko Robnik-Sikonja,et al.  An adaptation of Relief for attribute estimation in regression , 1997, ICML.

[62]  Wei Li,et al.  Information Extraction Supported Question Answering , 1999, TREC.

[63]  Ulf Hermjakob,et al.  Parsing and Question Classification for Question Answering , 2001, ACL 2001.

[64]  Leonard E. Trigg,et al.  Naive Bayes for regression , 1998 .

[65]  Robert F. Simmons,et al.  Computational Linguistics Natural Language Question- Answering Systems: 1969 , 2022 .

[66]  Charles L. A. Clarke,et al.  Passage retrieval vs. document retrieval for factoid question answering , 2003, SIGIR.

[67]  Alistair Moffat,et al.  Exploring the similarity space , 1998, SIGF.

[68]  Robin Cooper,et al.  The syntax and semantics of when-questions , 1982 .

[69]  Peter Willett,et al.  The limitations of term co-occurrence data for query expansion in document retrieval systems , 1991, J. Am. Soc. Inf. Sci..

[70]  W. Lehnert Cognition, Computers, and Car Bombs: How Yale Prepared Me for the 90's , 2013 .

[71]  Jacques Savoy,et al.  Statistical inference in retrieval effectiveness evaluation , 1997, Inf. Process. Manag..

[72]  Ellen M. Voorhees,et al.  Overview of the TREC 2002 Question Answering Track , 2003, TREC.

[73]  Kui-Lam Kwok,et al.  TREC-9 Cross Language, Web and Question-Answering Track Experiments using PIRCS , 2000, TREC.

[74]  J. R. Quinlan Learning With Continuous Classes , 1992 .

[75]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[76]  Sanda M. Harabagiu,et al.  TextNet - A text-based intelligent system , 1997, Nat. Lang. Eng..

[77]  Kui-Lam Kwok,et al.  TREC2001 Question-Answer, Web and Cross Language Experiments using PIRCS , 2001, TREC.

[78]  Ellen M. Voorhees,et al.  Evaluating the Evaluation: A Case Study Using the TREC 2002 Question Answering Track , 2003, NAACL.

[79]  Ellen M. Voorhees,et al.  Building a question answering test collection , 2000, SIGIR '00.

[80]  Robert F. Simmons,et al.  Answering English questions by computer: a survey , 1965, CACM.

[81]  Michael Colclough The Process of Question Answering — A Computer Simulation of Cognition , 1979 .

[82]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[83]  Susan T. Dumais,et al.  An Analysis of the AskMSR Question-Answering System , 2002, EMNLP.

[84]  David A. Hull Stemming Algorithms: A Case Study for Detailed Evaluation , 1996, J. Am. Soc. Inf. Sci..

[85]  Debashis Kushary,et al.  Bootstrap Methods and Their Application , 2000, Technometrics.

[86]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[87]  Dragomir R. Radev,et al.  The Use of Predictive Annotation for Question Answering in TREC8 , 1999, TREC.

[88]  Salim Roukos,et al.  Automatic Derivation of Surface Text Patterns for a Maximum Entropy Based Question Answering System , 2003, NAACL.

[89]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[90]  S. Robertson The probability ranking principle in IR , 1997 .

[91]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[92]  Martin M. Soubbotin Patterns of Potential Answer Expressions as Clues to the Right Answers , 2001, TREC.

[93]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[94]  Dell Zhang,et al.  Question classification using support vector machines , 2003, SIGIR.

[95]  Kristian J. Hammond,et al.  Question Answering from Frequently Asked Question Files: Experiences with the FAQ FINDER System , 1997, AI Mag..

[96]  Helmut Schmidt,et al.  Probabilistic part-of-speech tagging using decision trees , 1994 .

[97]  Walter Daelemans,et al.  Complex answers: a case study using a WWW question answering system , 2001, Natural Language Engineering.

[98]  Jacques Savoy,et al.  Term Proximity Scoring for Keyword-Based Retrieval Systems , 2003, ECIR.

[99]  Ellen M. Voorhees,et al.  Overview of the TREC-9 Question Answering Track , 2000, TREC.

[100]  Antonio Cisternino,et al.  PiQASso: Pisa Question Answering System , 2001, TREC.

[101]  Marti A. Hearst Automated Discovery of WordNet Relations , 2004 .

[102]  William H. Press,et al.  Numerical recipes in C , 2002 .

[103]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[104]  Chris Buckley,et al.  New Retrieval Approaches Using SMART: TREC 4 , 1995, TREC.

[105]  Salim Roukos,et al.  IBM's Statistical Question Answering System-TREC 11 , 2001, TREC.

[106]  Robert F. Simmons,et al.  Indexing and dependency logic for answering english questions , 1964 .

[107]  Peter Thanisch,et al.  Natural language interfaces to databases – an introduction , 1995, Natural Language Engineering.

[108]  Philip H. Ramsey Nonparametric Statistical Methods , 1974, Technometrics.

[109]  Ellen M. Voorhees,et al.  Overview of the seventh text retrieval conference (trec-7) [on-line] , 1999 .

[110]  David R. Musser,et al.  STL tutorial and reference guide - C++ programming with the standard template library , 1996, Addison-Wesley professional computing series.

[111]  Luis Gravano,et al.  Learning search engine specific query transformations for question answering , 2001, WWW '01.

[112]  Clement T. Yu,et al.  An Evaluation of Term Dependence Models in Information Retrieval , 1982, SIGIR.

[113]  Eduard H. Hovy,et al.  Offline Strategies for Online Question Answering: Answering Questions Before They Are Asked , 2003, ACL.

[114]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[115]  Charles L. A. Clarke,et al.  Relevance ranking for one to three term queries , 1997, Inf. Process. Manag..

[116]  David A. Hull Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[117]  Jamie Callan,et al.  Passage-retrieval evidence in document retrieval , 1994, SIGIR 1994.

[118]  Gerard Salton,et al.  Automatic indexing , 1980, ACM '80.

[119]  Adwait Ratnaparkhi,et al.  A Maximum Entropy Approach to Identifying Sentence Boundaries , 1997, ANLP.

[120]  Bertram Raphael SIR: A COMPUTER PROGRAM FOR SEMANTIC INFORMATION RETRIEVAL , 1964 .

[121]  A. Graesser,et al.  Mechanisms that generate questions , 1992 .

[122]  William A. Woods,et al.  Computational Linguistics Transition Network Grammars for Natural Language Analysis , 2022 .

[123]  James Allan,et al.  Automatic Query Expansion Using SMART: TREC 3 , 1994, TREC.

[124]  Peter Mark Roget,et al.  Roget's International Thesaurus , 1977 .

[125]  木村 和夫 Pragmatics , 1997, Language Teaching.

[126]  Anthony C. Davison,et al.  The Bootstrap , 2020 .

[127]  William S. Cooper,et al.  Some inconsistencies and misidentified modeling assumptions in probabilistic information retrieval , 1995, TOIS.

[128]  Maarten de Rijke,et al.  The University of Amsterdam at CLEF 2003 , 2001, CLEF.

[129]  Patrick Pantel,et al.  Discovery of inference rules for question-answering , 2001, Natural Language Engineering.

[130]  Ian H. Witten,et al.  Induction of model trees for predicting continuous classes , 1996 .

[131]  Charles L. A. Clarke,et al.  Question Answering by Passage Selection (MultiText Experiments for TREC-9) , 2000, TREC.

[132]  Mark Sanderson,et al.  Universities of Leeds, Sheffield and York http://eprints.whiterose.ac.uk/ , 2022 .

[133]  Toni Rietveld,et al.  Statistical Techniques for the Study of Language and Language Behaviour , 1993 .

[134]  C. J. van Rijsbergen,et al.  Report on the need for and provision of an 'ideal' information retrieval test collection , 1975 .

[135]  Piek Vossen,et al.  EuroWordNet: A multilingual database with lexical semantic networks , 1998, Springer Netherlands.

[136]  J. P. Thorne,et al.  Automatic Language Analysis , 1961 .

[137]  Hongbo Xu,et al.  ICT Experiments in TREC 11 QA Main Task , 2002, TREC.

[138]  Maarten de Rijke,et al.  Shallow Morphological Analysis in Monolingual Information Retrieval for Dutch, German, and Italian , 2001, CLEF.

[139]  Fernando Llopis,et al.  Passage Selection to Improve Question Answering , 2002, COLING 2002.

[140]  Chris Buckley,et al.  Improving automatic query expansion , 1998, SIGIR '98.

[141]  M. de Rijke,et al.  Tequesta: The University of Amsterdam's Textual Question Answering System , 2001, TREC.

[142]  Grace Hui Yang,et al.  Structured use of external knowledge for event-based open domain question answering , 2003, SIGIR.

[143]  Marko Robnik-Sikonja,et al.  Theoretical and Empirical Analysis of ReliefF and RReliefF , 2003, Machine Learning.

[144]  Chris Buckley,et al.  SMART in TREC 8 , 1999, Text Retrieval Conference.

[145]  Richard M. Schwartz,et al.  An Algorithm that Learns What's in a Name , 1999, Machine Learning.

[146]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[147]  Jimmy J. Lin,et al.  Gathering Knowledge for a Question Answering System from Heterogeneous Information Sources , 2001, HTLKM@ACL.

[149]  Jun Suzuki,et al.  Question Classification using HDAG Kernel , 2003, ACL 2003.

[150]  Ellen M. Voorhees,et al.  Query expansion using lexical-semantic relations , 1994, SIGIR '94.

[151]  Fernando Llopis,et al.  University of Alicante Experiments at TREC 2002 , 2002, TREC.

[152]  Remko J. H. Scha,et al.  Philips question-answering system PHLIQA1 , 1977, SGAR.

[153]  Alain Colmerauer,et al.  Metamorphosis Grammars , 1978, Natural Language Communication with Computers.

[154]  Donna K. Harman,et al.  Overview of the Sixth Text REtrieval Conference (TREC-6) , 1997, Inf. Process. Manag..

[155]  Ellen M. Voorhees,et al.  The effect of topic set size on retrieval experiment error , 2002, SIGIR '02.

[156]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[157]  Usama M. Fayyad,et al.  Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning , 1993, IJCAI.

[158]  Justin Zobel,et al.  Effective ranking with arbitrary passages , 2001, J. Assoc. Inf. Sci. Technol..

[159]  Scott Miller,et al.  TREC 2002 QA at BBN: Answer Selection and Confidence Estimation , 2002, TREC.

[160]  Stephen E. Robertson,et al.  Okapi at TREC-7: Automatic Ad Hoc, Filtering, VLC and Interactive , 1998, TREC.

[161]  Christian Jacquemin,et al.  QALC--The Question-Answering System of LIMSI-CNRS , 2000, TREC.

[162]  Karen Spärck Jones Automatic language and information processing: rethinking evaluation , 2001, Natural Language Engineering.

[163]  R. Scha Logical foundations for question answering , 1983 .

[164]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[165]  W. Bruce Croft,et al.  Query expansion using local and global document analysis , 1996, SIGIR '96.

[166]  Ian H. Witten,et al.  Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[167]  Sanda M. Harabagiu,et al.  Performance issues and error analysis in an open-domain question answering system , 2003, TOIS.

[168]  Wessel Kraaij,et al.  Viewing stemming as recall enhancement , 1996, SIGIR '96.

[169]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[170]  L. A. Miller The Process of Question Answering - A Computer Simulation of Cognition , 1980, CL.

[171]  Eduard H. Hovy,et al.  Learning surface text patterns for a Question Answering System , 2002, ACL.

[172]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[173]  Gerard Salton,et al.  Document Length Normalization , 1995, Inf. Process. Manag..

[174]  Eduard H. Hovy,et al.  Question Answering in Webclopedia , 2000, TREC.

[175]  Jong-Hyeok Lee,et al.  Question Answering Approach Using a WordNet-based Answer Type Taxonomy , 2002, TREC.

[176]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[177]  M. Kendall A NEW MEASURE OF RANK CORRELATION , 1938 .

[178]  Robert J. Gaizauskas,et al.  The University of Sheffield TREC 2002 Q&A System , 2002, TREC.

[179]  T. Gonen,et al.  Questions , 1927, Journal of Family Planning and Reproductive Health Care.

[180]  Jimmy J. Lin,et al.  Quantitative evaluation of passage retrieval algorithms for question answering , 2003, SIGIR.

[181]  Fredric C. Gey,et al.  Full Text Retrieval based on Probalistic Equations with Coefficients fitted by Logistic Regression , 1993, TREC.

[182]  Stefanie Tellex,et al.  Pauchok: A Modular Framework for Question Answering , 2003 .

[183]  W. John Wilbur,et al.  Non-parametric significance tests of retrieval performance comparisons , 1994, J. Inf. Sci..

[184]  Justin Zobel,et al.  Passage retrieval revisited , 1997, SIGIR '97.

[185]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[186]  J. Ginzburg Interrogatives: questions, facts and dialogue , 1996 .

[187]  Dragomir R. Radev,et al.  Question-answering by predictive annotation , 2000, SIGIR '00.

[188]  Luis Gravano,et al.  Learning to find answers to questions on the Web , 2004, TOIT.

[189]  Grace Hui Yang,et al.  The Integration of Lexical Knowledge and External Resources for Question Answering , 2002, TREC.

[190]  Jennifer Chu-Carroll,et al.  A Multi-Strategy and Multi-Source Approach to Question Answering , 2002, TREC.

[191]  Bernardo Magnini,et al.  Exploiting Lexical Expansions and Boolean Compositions for Web Querying , 2000 .

[192]  James R. Driscoll,et al.  Incorporating a semantic analysis into a document retrieval strategy , 1991, SIGIR '91.