How to account for mispellings: Quantifying the benefit of character representations in neural content scoring models

Character-based representations in neural models have been claimed to be a tool to overcome spelling variation in in word token-based input. We examine this claim in neural models for content scoring. We formulate precise hypotheses about the possible effects of adding character representations to word-based models and test these hypotheses on large-scale real world content scoring datasets. We find that, while character representations may provide small performance gains in general, their effectiveness in accounting for spelling variation may be limited. We show that spelling correction can provide larger gains than character representations, and that spelling correction improves the performance of models with character representations. With these insights, we report a new state of the art on the ASAP-SAS content scoring dataset.

[1]  Benno Stein,et al.  The Eras and Trends of Automatic Short Answer Grading , 2015, International Journal of Artificial Intelligence in Education.

[2]  Radu Tudor Ionescu,et al.  Automated essay scoring with string kernels and word embeddings , 2018, ACL.

[3]  Hwee Tou Ng,et al.  A Neural Approach to Automated Essay Scoring , 2016, EMNLP.

[4]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[5]  Shujian Huang,et al.  Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention , 2018, NAACL.

[6]  Torsten Zesch,et al.  The Influence of Spelling Errors on Content Scoring Performance , 2017, NLP-TEA@IJCNLP.

[7]  Peter W. Foltz,et al.  Identifying Patterns For Short Answer Scoring Using Graph-based Lexico-Semantic Text Matching , 2015, BEA@NAACL-HLT.

[8]  Mark D. Shermis,et al.  Contrasting State-of-the-Art in the Machine Scoring of Short-Form Constructed Responses , 2015 .

[9]  Christopher D. Manning,et al.  Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models , 2016, ACL.

[10]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[11]  Shourya Roy,et al.  Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading , 2017, IJCAI.

[12]  Yue Zhang,et al.  Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring , 2017, CoNLL.

[13]  Torsten Zesch,et al.  The Influence of Variance in Learner Answers on Automatic Content Scoring , 2019, Front. Educ..

[14]  Tong Zhang,et al.  Deep Pyramid Convolutional Neural Networks for Text Categorization , 2017, ACL.

[15]  Bikram Sengupta,et al.  Creating Scoring Rubric from Representative Student Answers for Improved Short Answer Grading , 2018, CIKM.

[16]  Tong Zhang,et al.  Effective Use of Word Order for Text Categorization with Convolutional Neural Networks , 2014, NAACL.

[17]  Richard Inger,et al.  A brief introduction to mixed effects modelling and multi-model inference in ecology , 2018, PeerJ.

[18]  Bikram Sengupta,et al.  Sentence Level or Token Level Features for Automatic Short Answer Grading?: Use Both , 2018, AIED.

[19]  Anastassia Loukina,et al.  A Large Scale Quantitative Exploration of Modeling Strategies for Content Scoring , 2017, BEA@EMNLP.

[20]  Torsten Zesch,et al.  Investigating neural architectures for short answer scoring , 2017, BEA@EMNLP.

[21]  Guoyin Wang,et al.  Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms , 2018, ACL.

[22]  Michael Flor,et al.  Four types of context for automatic spelling correction , 2012, TAL.

[23]  Swati Aggarwal,et al.  Get IT Scored Using AutoSAS - An Automated System for Scoring Short Answers , 2019, AAAI.

[24]  Michael Flor,et al.  On using context for automatic correction of non-word misspellings in student essays , 2012, BEA@NAACL-HLT.

[25]  Chandra Bhagavatula,et al.  Semi-supervised sequence tagging with bidirectional language models , 2017, ACL.