Integrating parallel analysis modules to evaluate the meaning of answers to reading comprehension questions

Contextualised, meaning-based interaction in the foreign language is widely recognised as crucial for second language acquisition. Correspondingly, current exercises in foreign language teaching generally require students to manipulate both form and meaning. For intelligent language tutoring systems to support such activities, they thus must be able to evaluate the appropriateness of the meaning of a learner response for a given exercise. We discuss such a content-assessment approach, focusing on reading comprehension exercises. We pursue the idea that a range of simultaneously available representations at different levels of complexity and linguistic abstraction provide a good empirical basis for content assessment. We show how an annotation-based NLP architecture implementing this idea can be realised and that it successfully performs on a corpus of authentic learner answers to reading comprehension questions. To support comparison and sustainable development on content assessment, we also define a general exchange format for such exercise data.

[1]  Eleazar Eskin,et al.  Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Combinations via Machine Learning , 1999, EMNLP.

[2]  R. Ellis Instructed Language Learning and Task-Based Teaching , 2005 .

[3]  Walt Detmar Meurers Diagnosing Meaning Errors in Short Answers to Reading Comprehension Questions , 2008 .

[4]  Judith Westphal Irwin,et al.  Teaching reading comprehension processes , 1986 .

[5]  Dan Roth,et al.  Learning Question Classifiers , 2002, COLING.

[6]  Walt Detmar Meurers,et al.  Analyzing Learner Language : Towards A Flexible NLP Architecture for Intelligent Language Tutors , 2010 .

[7]  Claudia Leacock Scoring Free-Responses Automatically: A Case Study of a Large-Scale Assessment , 2004 .

[8]  Manfred Krifka,et al.  Basic notions of information structure , 2008 .

[9]  Christopher D. Manning LOCAL TEXTUAL INFERENCE : IT'S HARD TO CIRCUMSCRIBE , BUT YOU KNOW IT WHEN YOU SEE IT - AND NLP NEEDS IT , 2006 .

[10]  Eli Hinkel,et al.  Handbook of Research in Second Language Teaching and Learning : Volume 2 , 2011 .

[11]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[12]  Janet S. Twyman,et al.  Teaching Reading Comprehension , 1978 .

[13]  L. S. Shapley,et al.  College Admissions and the Stability of Marriage , 2013, Am. Math. Mon..

[14]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[15]  Diana Pérez-Marín Adaptive Computer Assisted Assessment of free-text students' answers: An approach to automatically generate students' conceptual models , 2009 .

[16]  Chris Brockett,et al.  Support Vector Machines for Paraphrase Identification and Corpus Construction , 2005, IJCNLP.

[17]  Walter Daelemans,et al.  TiMBL: Tilburg Memory-Based Learner , 2007 .

[18]  Cheryl L. Champeau de Lopez,et al.  Taxonomy: Evaluating Reading Comprehension in EFL. , 1997 .

[19]  Stacey Bailey,et al.  Content Assessment in Intelligent Computer-aided Language Learning: Meaning Error Diagnosis for English as a Second Language , 2008 .

[20]  Thilo Götz,et al.  Design and implementation of the UIMA Common Analysis System , 2004, IBM Syst. J..

[21]  Elliotte Rusty Harold,et al.  XML in a Nutshell , 2001 .

[22]  Walt Detmar Meurers Compiling a Task-Based Corpus for the Analysis of Learner Language in Context , 2009 .

[23]  Alon Lavie,et al.  Extending the METEOR Machine Translation Evaluation Metric to the Phrase Level , 2010, NAACL.

[24]  Ido Dagan,et al.  Recognizing textual entailment: Rational, evaluation and approaches , 2009, Natural Language Engineering.

[25]  David A. Ferrucci,et al.  UIMA: an architectural approach to unstructured information processing in the corporate research environment , 2004, Natural Language Engineering.