Recognizing entailment in intelligent tutoring systems*

Abstract This paper describes a new method for recognizing whether a student's response to an automated tutor's question entails that they understand the concepts being taught. We demonstrate the need for a finer-grained analysis of answers than is supported by current tutoring systems or entailment databases and describe a new representation for reference answers that addresses these issues, breaking them into detailed facets and annotating their entailment relationships to the student's answer more precisely. Human annotation at this detailed level still results in substantial interannotator agreement (86.2%), with a kappa statistic of 0.728. We also present our current efforts to automatically assess student answers, which involves training machine learning classifiers on features extracted from dependency parses of the reference answer and student's response and features derived from domain-independent lexical statistics. Our system's performance, as high as 75.5% accuracy within domain and 68.8% out of domain, is very encouraging and confirms the approach is feasible. Another significant contribution of this work is that it represents a significant step in the direction of providing domain-independent semantic assessment of answers. No prior work in the area of tutoring or educational assessment has attempted to build such domain-independent systems. They have virtually all required hundreds of examples of learner answers for each new question in order to train aspects of their systems or to hand-craft information extraction templates.

[1]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[2]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[3]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[4]  Lawrence F. Lowery,et al.  Full Option Science System(FOSS) , 1998 .

[5]  Martha Palmer,et al.  Class-Based Construction of a Verb Lexicon , 2000, AAAI/IAAI.

[6]  Daniel Gildea,et al.  Automatic Labeling of Semantic Roles , 2000, ACL.

[7]  Luis Gravano,et al.  Snowball: extracting relations from large plain-text collections , 2000, DL '00.

[8]  Vincent Aleven,et al.  A Tutorial Dialogue System with Knowledge-Based Understanding and Classification of Student Explanations , 2001 .

[9]  David Callear,et al.  CAA of Short Non-MCQ Answers , 2001 .

[10]  Regina Barzilay,et al.  Extracting Paraphrases from a Parallel Corpus , 2001, ACL.

[11]  Ralph Grishman,et al.  Automatic Pattern Acquisition for Japanese Information Extraction , 2001, HLT.

[12]  Patrick Pantel,et al.  Discovery of inference rules for question-answering , 2001, Natural Language Engineering.

[13]  Peter D. Turney.  Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL , 2001, ECML.

[14]  Tom Mitchell,et al.  Towards robust computerised marking of free-text responses , 2002 .

[15]  Eduard H. Hovy,et al.  Learning surface text patterns for a Question Answering System , 2002, ACL.

[16]  Regina Barzilay,et al.  Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment , 2003, NAACL.

[17]  Stephen Pulman,et al.  Automarking: using computational linguistics to score short‚ free−text responses , 2003 .

[18]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[19]  Carolyn Penstein Rosé,et al.  A Hybrid Text Classification Approach for Analysis of Student Essays , 2003, HLT-NAACL 2003.

[20]  Oren Glickman.  IDENTIFYING LEXICAL PARAPHRASES FROM A SINGLE CORPUS: A CASE STUDY FOR VERBS , 2003 .

[21]  K. VanLehn,et al.  Abductive Theorem Proving for Analyzing Student Explanations , 2003 .

[22]  Daniel Marcu,et al.  Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences , 2003, NAACL.

[23]  Jonathan Ginzburg,et al.  Proceedings of COLING 2004 , 2004 .

[24]  Chris Quirk,et al.  Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources , 2004, COLING.

[25]  Kurt VanLehn,et al.  Abductive Theorem Proving for Analyzing Student Explanations to Guide Feedback in Intelligent Tutoring Systems , 2004, Journal of Automated Reasoning.

[26]  Kurt VanLehn,et al.  Abductive Theorem Proving for Analyzing Student Explanations to Guide Feedback in Intelligent Tutoring Systems , 2004 .

[27]  Stanley Peters,et al.  Contextualizing learning in a reflective conversational tutor , 2004, IEEE International Conference on Advanced Learning Technologies, 2004. Proceedings..

[28]  Yuji Matsumoto.  MaltParser: A language-independent system for data-driven dependency parsing , 2005 .

[29]  John D. Burger,et al.  Generating an Entailment Corpus from News Headlines , 2005, EMSEE@ACL.

[30]  Ido Dagan,et al.  Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment@ACL 2005, Ann Arbor, Michigan, USA, June 30, 2005 , 2005, EMSEE@ACL.

[31]  Lucy Vanderwende,et al.  What Syntax Can Contribute in the Entailment Task , 2005, MLCW.

[32]  Dan Roth,et al.  An Inference Model for Semantic Entailment in Natural Language , 2005, IJCAI.

[33]  Ido Dagan,et al.  Web Based Probabilistic Textual Entailment , 2005 .

[34]  Roy Bar-Haim,et al.  Definition and Analysis of Intermediate Entailment Levels , 2005, EMSEE@ACL.

[35]  Christopher D. Manning,et al.  Robust Textual Inference using Diverse Knowledge Sources , 2005 .

[36]  Siobhan Chapman.  Logic and Conversation , 2005 .

[37]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[38]  Rodney D. Nielsen,et al.  Toward Dependency Path based Entailment , 2006 .

[39]  Christopher D. Manning,et al.  Learning to recognize features of valid textual entailments , 2006, NAACL.

[40]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[41]  Andrew Hickl,et al.  A Discourse Commitment-Based Framework for Recognizing Textual Entailment , 2007, ACL-PASCAL@ACL.

[42]  Dan I. Moldovan,et al.  COGEX at RTE 3 , 2007, ACL-PASCAL@ACL.

[43]  Rodney D. Nielsen,et al.  A Corpus of Fine-Grained Entailment Relations , 2007, ACL-PASCAL@ACL.

[44]  Kentaro Inui,et al.  Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing , 2007, ACL 2007.

[45]  Rodney D. Nielsen,et al.  Soft Computing in Intelligent Tutoring Systems and Educational Assessment , 2008, Soft Computing Applications in Business.

[46]  Bhanu Prasad.  Soft Computing Applications in Business , 2008, Soft Computing Applications in Business.