Generating Diagnostic Multiple Choice Comprehension Cloze Questions

This paper describes and evaluates DQGen, which automatically generates multiple choice cloze questions to test a child's comprehension while reading a given text. Unlike previous methods, it generates different types of distracters designed to diagnose different types of comprehension failure, and tests comprehension not only of an individual sentence but of the context that precedes it. We evaluate the quality of the overall questions and the individual distracters, according to 8 human judges blind to the correct answers and intended distracter types. The results, errors, and judges' comments reveal limitations and suggest how to address some of them.

[1]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[2]  Caroline Sporleder,et al.  Topic Models for Word Sense Disambiguation and Token-Based Idiom Detection , 2010, ACL.

[3]  Maxine Eskénazi,et al.  Automatic Question Generation for Vocabulary Assessment , 2005, HLT.

[4]  M. Kendall,et al.  The Problem of $m$ Rankings , 1939 .

[5]  Tomoko Kojiri,et al.  Automatic Generation System of Multiple-Choice Cloze Questions and its Evaluation , 2010 .

[6]  Donna Marie Gates.  How to Generate Cloze Questions from Definitions: A Syntactic Approach , 2011, AAAI Fall Symposium: Question Generation.

[7]  Judy Kay,et al.  Proceedings of the 15th international conference on Artificial intelligence in education , 2011 .

[8]  I. Trancoso,et al.  Automatic Generation of Cloze Question Distractors , 2010 .

[9]  Walter Kintsch,et al.  An Overview of Top-Down and Bottom-Up Effects in Comprehension: The CI Perspective , 2005 .

[10]  Montse Maritxalar,et al.  A Study on the Automatic Selection of Candidate Sentences Distractors , 2009, AIED.

[11]  Jack Mostow,et al.  Using Automated Questions to Assess Reading Comprehension, Vocabulary, and Effects of Tutorial Interventions , 2004 .

[12]  Chao-Lin Liu,et al.  Using Lexical Constraints to Enhance the Quality of Computer-Generated Multiple-Choice Cloze Items , 2005, Int. J. Comput. Linguistics Chin. Lang. Process..

[13]  Jack Mostow,et al.  Can a Computer Listen for Fluctuations in Reading Comprehension? , 2007, AIED.

[14]  Steven A. Stahl,et al.  Children's reading comprehension and assessment , 2005 .

[15]  Dan Klein,et al.  Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network , 2003, NAACL.

[16]  Paul van den Broek,et al.  6. Comprehension and memory of science texts: inferential processes and the construction of a mental representantion , 2002 .

[17]  Jack Mostow,et al.  Toward Exploiting EEG Input in a Reading Tutor , 2013, Int. J. Artif. Intell. Educ..

[18]  Caroline Sporleder,et al.  Classifier Combination for Contextual Idiom Detection Without Labelled Data , 2009, EMNLP.

[19]  Eiichiro Sumita,et al.  Measuring Non-native Speakers’ Proficiency of English by Using a Test with Automatically-Generated Fill-in-the-Blank Questions , 2005 .

[20]  Arthur C. Graesser,et al.  The psychology of science text comprehension , 2014 .

[21]  David Coniam.  A Preliminary Inquiry into Using Corpus Word Frequency Data in the Automatic Generation of English Language Cloze Tests , 2013 .

[22]  Walter Kintsch,et al.  Information accretion and reduction in text processing: Inferences , 1993 .

[23]  Darielle Greenberg.  Woodcock Reading Mastery Tests–Revised , 2014 .

[24]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[25]  Arthur C. Graesser,et al.  The Construction of Causal Inferences While Reading Expository Texts on Science and Technology , 1998 .

[26]  Michael Heilman,et al.  A Selection Strategy to Improve Cloze Question Quality , 2008 .

[27]  史尚明.  Learning words right with the Sketch Engine and WebBootCat: Automatic cloze generation from corpora and the web , 2008 .

[28]  Daniel Dominic Sleator,et al.  Parsing English with a Link Grammar , 1995, IWPT.

[29]  Joseph E. Beck,et al.  Better Student Assessing by Finding Difficulty Factors in a Fully Automated Comprehension Measure , 2006, Intelligent Tutoring Systems.

[30]  Montse Maritxalar,et al.  Automatic Distractor Generation for Domain Specific Texts , 2010, IceTAL.

[31]  L.T.W. Verhoeven,et al.  Interactive literacy education: facilitating literacy environments through technology , 2008 .

[32]  Chao-Lin Liu,et al.  Applications of Lexical Information for Algorithmically Composing Multiple-Choice Cloze Items , 2005 .