Answer Extraction as Sequence Tagging with Tree Edit Distance

Our goal is to extract answers from preretrieved sentences for Question Answering (QA). We construct a linear-chain Conditional Random Field based on pairs of questions and their possible answer sentences, learning the association between questions and answer types. This casts answer extraction as an answer sequence tagging problem for the first time, where knowledge of shared structure between question and source sentence is incorporated through features based on Tree Edit Distance (TED). Our model is free of manually created question and answer templates, fast to run (processing 200 QA pairs per second excluding parsing time), and yields an F1 of 63.3% on a new public dataset based on prior TREC QA evaluations. The developed system is open-source, and includes an implementation of the TED model that is state of the art in the task of ranking QA pairs.

[1]  M. Felisa Verdejo,et al.  Overview of the Answer Validation Exercise 2007 , 2006, CLEF.

[2]  Denilson Barbosa,et al.  TASM: Top-k Approximate Subtree Matching , 2010, 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010).

[3]  Philippe Mulhem,et al.  LIG at ImageCLEF 2008, Evaluating Systems for Multilingual and Multimodal Information Access , 2008 .

[4]  Alberto H. F. Laender,et al.  Automatic web news extraction using tree edit distance , 2004, WWW '04.

[5]  Noah A. Smith,et al.  What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA , 2007, EMNLP.

[6]  Ni Lao,et al.  Complex Cross-lingual Question Answering as a Sequential Classification and Multi-Document Summarization Task , 2008, NTCIR.

[7]  James R. Curran,et al.  Question classification with log-linear models , 2006, SIGIR.

[8]  Philip Bille,et al.  A survey on tree edit distance and related problems , 2005, Theor. Comput. Sci..

[9]  Peter Clark,et al.  Enabling domain experts to convey questions to a machine: a modified, template-based approach , 2003, K-CAP '03.

[10]  Noah A. Smith,et al.  Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions , 2010, NAACL.

[11]  M. Felisa Verdejo,et al.  Testing the Reasoning for Question Answering Validation , 2008, J. Log. Comput..

[12]  Dell Zhang,et al.  Question classification using support vector machines , 2003, SIGIR.

[13]  Roberto Basili,et al.  Exploiting Syntactic and Shallow Semantic Kernels for Question Answer Classification , 2007, ACL.

[14]  Kaizhong Zhang,et al.  Simple Fast Algorithms for the Editing Distance Between Trees and Related Problems , 1989, SIAM J. Comput..

[15]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[16]  Xiaoyan Zhu,et al.  Using Conditional Random Fields to Extract Contexts and Answers of Questions from Online Forums , 2008, ACL.

[17]  B. Magnini,et al.  Recognizing Textual Entailment with Tree Edit Distance Algorithms , 2005 .

[18]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[19]  Jennifer Chu-Carroll,et al.  Building Watson: An Overview of the DeepQA Project , 2010, AI Mag..

[20]  Salim Roukos,et al.  Automatic Derivation of Surface Text Patterns for a Maximum Entropy Based Question Answering System , 2003, NAACL.

[21]  Martin M. Soubbotin.  Patterns of Potential Answer Expressions as Clues to the Right Answers , 2001, TREC.

[22]  Sanda M. Harabagiu,et al.  Performance issues and error analysis in an open-domain question answering system , 2003, TOIS.

[23]  Bernardo Magnini,et al.  Is It the Right Answer? Exploiting Web Redundancy for Answer Validation , 2002, ACL.

[24]  W. Bruce Croft,et al.  Analysis of Statistical Question Classification for Fact-Based Questions , 2005, Information Retrieval.

[25]  David A. Smith,et al.  Quasi-Synchronous Grammars: Alignment by Soft Projection of Syntactic Dependencies , 2006, WMT@HLT-NAACL.

[26]  Achim G. Hoffmann,et al.  Proceedings of the Nineteenth International Conference on Machine Learning , 2002 .

[27]  Christopher D. Manning,et al.  Probabilistic Tree-Edit Models with Structured Latent Variables for Textual Entailment and Question Answering , 2010, COLING.

[28]  Dan Roth,et al.  Learning Question Classifiers , 2002, COLING.

[29]  Eriks Sneiders,et al.  Automated question answering : template-based approach , 2002 .

[30]  Richard M. Schwartz,et al.  An Algorithm that Learns What's in a Name , 1999, Machine Learning.

[31]  Ulf Hermjakob,et al.  Parsing and Question Classification for Question Answering , 2001, ACL 2001.

[32]  Koby Crammer,et al.  Online Large-Margin Training of Dependency Parsers , 2005, ACL.

[33]  Wayne H. Ward,et al.  Question Classification with Support Vector Machines and Error Correcting Codes , 2003, HLT-NAACL.

[34]  Adwait Ratnaparkhi,et al.  A Maximum Entropy Model for Part-Of-Speech Tagging , 1996, EMNLP.

[35]  Dan Roth,et al.  Mapping Dependencies Trees: An Application to Question Answering , 2003 .

[36]  James Allan,et al.  A comparison of statistical significance tests for information retrieval evaluation , 2007, CIKM '07.