Evaluating Textual Entailment Recognition for University Entrance Examinations

The present article addresses an attempt to apply questions in university entrance examinations to the evaluation of textual entailment recognition. Questions in several fields, such as history and politics, primarily test the examinee’s knowledge in the form of choosing true statements from multiple choices. Answering such questions can be regarded as equivalent to finding evidential texts from a textbase such as textbooks and Wikipedia. Therefore, this task can be recast as recognizing textual entailment between a description in a textbase and a statement given in a question. We focused on the National Center Test for University Admission in Japan and converted questions into the evaluation data for textual entailment recognition by using Wikipedia as a textbase. Consequently, it is revealed that nearly half of the questions can be mapped into textual entailment recognition; 941 text pairs were created from 404 questions from six subjects. This data set is provided for a subtask of NTCIR RITE (Recognizing Inference in Text), and 16 systems from six teams used the data set for evaluation. The evaluation results revealed that the best system achieved a correct answer ratio of 56%, which is significantly better than a random choice baseline.

[1]  Marcello Federico,et al.  Using Bilingual Parallel Corpora for Cross-Lingual Textual Entailment , 2011, ACL.

[2]  Peter Clark,et al.  The Seventh PASCAL Recognizing Textual Entailment Challenge , 2011, TAC.

[3]  Ido Dagan,et al.  The Sixth PASCAL Recognizing Textual Entailment Challenge , 2009, TAC.

[4]  Steffen Staab,et al.  Ontology-Based Query and Answering in Chemistry: OntoNova @ Project Halo , 2003, SEMWEB.

[5]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[6]  Eduard H. Hovy,et al.  Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation , 2011, CLEF.

[7]  N. H. Beebe.  A Complete Bibliography of ACM Transactions on Asian Language Information Processing , 2007 .

[8]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[9]  Sadao Kurohashi,et al.  Predicate-argument Structure based Textual Entailment Recognition System of KYOTO Team for NTCIR9 RITE , 2011, NTCIR.

[10]  Johan Bos,et al.  Textual Entailment at EVALITA 2009 , 2009 .

[11]  Yotaro Watanabe,et al.  TU Group at NTCIR9-RITE: Leveraging Diverse Lexical Resources for Recognizing Textual Entailment , 2011, NTCIR.

[12]  Shuming Shi,et al.  Overview of NTCIR-9 RITE: Recognizing Inference in TExt , 2011, NTCIR.

[13]  Sivaji Bandyopadhyay,et al.  A Textual Entailment System using Web based Machine Translation System , 2011, NTCIR.

[14]  Teruko Mitamura,et al.  LTI's Textual Entailment Recognizer System at NTCIR-9 RITE , 2011, NTCIR.

[15]  Ion Androutsopoulos,et al.  A Survey of Paraphrasing and Textual Entailment Methods , 2009, J. Artif. Intell. Res..

[16]  Akira Shimazu,et al.  A Machine Learning based Textual Entailment Recognition System of JAIST Team for NTCIR9 RITE , 2011, NTCIR.

[17]  Marcello Federico,et al.  Towards Cross-Lingual Textual Entailment , 2010, NAACL.

[18]  Ido Dagan,et al.  The Fourth PASCAL Recognizing Textual Entailment Challenge , 2008, TAC.

[19]  Roy Bar-Haim,et al.  The Second PASCAL Recognising Textual Entailment Challenge , 2006 .

[20]  Heng Ji,et al.  Knowledge Base Population: Successful Approaches and Challenges , 2011, ACL.

[21]  Hiroshi Kanayama,et al.  Syntactic Difference Based Approach for NTCIR-9 RITE Task , 2011, NTCIR.