Predicting Question Quality Using Recurrent Neural Networks

This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.

[1]  Christopher Meek,et al.  Semantic Parsing for Single-Relation Question Answering , 2014, ACL.

[2]  John D. Lafferty,et al.  Inducing Features of Random Fields , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  A. Graesser,et al.  Anomalous Information Triggers Questions When Adults Solve Quantitative Problems and Comprehend Stories. , 1993 .

[4]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[5]  Danielle S. McNamara,et al.  iSTART: A Web-based tutor that teaches self-explanation and metacognitive reading strategies. , 2007 .

[6]  Megha Mishra,et al.  Question Classification using Semantic, Syntactic and Lexical features , 2013 .

[7]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[8]  Arthur C. Graesser,et al.  Question-asking in advanced distributed learning environments. , 2007 .

[9]  Soumen Chakrabarti,et al.  Enhanced Answer Type Inference from Questions using Sequential Models , 2005, HLT/EMNLP.

[10]  Dell Zhang,et al.  Question classification using support vector machines , 2003, SIGIR.

[11]  Sanda M. Harabagiu,et al.  FALCON: Boosting Knowledge for Answer Engines , 2000, TREC.

[12]  Adwait Ratnaparkhi,et al.  IBM's Statistical Question Answering System , 2000, TREC.

[13]  Dan Roth,et al.  Learning Question Classifiers , 2002, COLING.

[14]  Danielle S. McNamara.  Reading comprehension strategies : theories, interventions, and technologies , 2007 .

[15]  Richard Socher,et al.  A Neural Network for Factoid Question Answering over Paragraphs , 2014, EMNLP.

[16]  A. Graesser,et al.  Question Asking During Tutoring , 1994 .

[17]  SchmidhuberJürgen,et al.  2005 Special Issue , 2005 .

[18]  Ulf Hermjakob,et al.  Parsing and Question Classification for Question Answering , 2001, ACL 2001.

[19]  Catherine Snow,et al.  Reading for Understanding: Toward an R&D Program in Reading Comprehension , 2002 .

[20]  Benjamin Bergen,et al.  Finding Non-Arbitrary Form-Meaning Systematicity Using String-Metric Learning for Kernel Regression , 2016, ACL.

[21]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[22]  Annemarie Sullivan Palincsar,et al.  Interactive Teaching to Promote Independent Learning from Text. , 1986 .

[23]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[24]  Richard Socher,et al.  Dynamic Coattention Networks For Question Answering , 2016, ICLR.

[25]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[26]  Eduard H. Hovy,et al.  Toward Semantics-Based Answer Pinpointing , 2001, HLT.

[27]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[28]  Arthur C. Graesser,et al.  Overhearing Dialogues and Monologues in Virtual Tutoring Sessions: Effects on Questioning and Vicarious Learning , 2000 .

[29]  Bowen Zhou,et al.  Attentive Pooling Networks , 2016, ArXiv.

[30]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[31]  B. Rosenshine,et al.  Teaching Students to Generate Questions: A Review of the Intervention Studies , 1996 .

[32]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[33]  Danielle S. McNamara,et al.  Assessing Question Quality Using NLP , 2017, AIED.

[34]  James R. Curran,et al.  Question classification with log-linear models , 2006, SIGIR.

[35]  Wei Jyh Heng,et al.  Question classification for e-learning by artificial neural network , 2003, Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia. Proceedings of the 2003 Joint.

[36]  B. Davey,et al.  Effects of question-generation training on reading comprehension , 1986 .

[37]  Phil Blunsom,et al.  A Convolutional Neural Network for Modelling Sentences , 2014, ACL.

[38]  Isabel L. Beck,et al.  Questioning the Author: An Approach for Enhancing Student Engagement With Text , 1997 .

[39]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[40]  Arthur C. Graesser,et al.  Utterance Classification in AutoTutor , 2003, HLT-NAACL 2003.

[41]  Wayne H. Ward,et al.  Question Classification with Support Vector Machines and Error Correcting Codes , 2003, HLT-NAACL.

[42]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[43]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[44]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[45]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[46]  Cheng Niu,et al.  Location Normalization for Information Extraction , 2002, COLING.

[47]  Luísa Coheur,et al.  From symbolic to sub-symbolic information in question classification , 2011, Artificial Intelligence Review.

[48]  Peter B. Mosenthal,et al.  Understanding the strategies of document literacy and their conditions of use. , 1996 .

[49]  Ali Farhadi,et al.  Bidirectional Attention Flow for Machine Comprehension , 2016, ICLR.

[50]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[51]  Jun Suzuki,et al.  Question Classification using HDAG Kernel , 2003, ACL 2003.

[52]  Traian Rebedea,et al.  Sentence selection with neural networks using string kernels , 2017, KES.

[53]  B. Rosenshine,et al.  Reciprocal Teaching: A Review of the Research , 1994 .