Automated essay scoring with string kernels and word embeddings

In this work, we present an approach based on combining string kernels and word embeddings for automatic essay scoring. String kernels capture the similarity among strings based on counting common character n-grams, which are a low-level yet powerful type of feature, demonstrating state-of-the-art results in various text classification tasks such as Arabic dialect identification or native language identification. To our best knowledge, we are the first to apply string kernels to automatically score essays. We are also the first to combine them with a high-level semantic feature representation, namely the bag-of-super-word-embeddings. We report the best performance on the Automated Student Assessment Prize data set, in both in-domain and cross-domain settings, surpassing recent state-of-the-art deep learning approaches.

[1]  Semire Dikli,et al.  An Overview of Automated Scoring of Essays. , 2006 .

[2]  Swapna Somasundaran,et al.  Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays , 2014, COLING.

[3]  Ben He,et al.  Automated Essay Scoring by Maximizing Human-Machine Agreement , 2013, EMNLP.

[4]  Jinhao Wang,et al.  Automated Essay Scoring versus Human Scoring: A Correlational Study. , 2008 .

[5]  Radu Tudor Ionescu,et al.  HASKER: An efficient algorithm for string kernels. Application to polarity classification in various languages , 2017, KES.

[6]  Radu Tudor Ionescu,et al.  Can string kernels pass the test of time in Native Language Identification? , 2017, BEA@EMNLP.

[7]  Jitendra Malik,et al.  Representing and Recognizing the Visual Appearance of Materials using Three-dimensional Textons , 2001, International Journal of Computer Vision.

[8]  Radu Tudor Ionescu,et al.  From Image to Text Classification: A Novel Approach based on Clustering Word Embeddings , 2017, KES.

[9]  Aoife Cahill,et al.  Can characters reveal your native language? A language-independent approach to native language identification , 2014, EMNLP.

[10]  Peter W. Foltz,et al.  Automated Essay Scoring: Applications to Educational Technology , 1999 .

[11]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[12]  Radu Tudor Ionescu,et al.  UnibucKernel: An Approach for Arabic Dialect Identification Based on Multiple String Kernels , 2016, VarDial@COLING.

[13]  Hwee Tou Ng,et al.  A Neural Approach to Automated Essay Scoring , 2016, EMNLP.

[14]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[15]  Leah S. Larkey,et al.  Automatic essay grading using text categorization techniques , 1998, SIGIR '98.

[16]  Hwee Tou Ng,et al.  Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression , 2015, EMNLP.

[17]  Paolo Rosso,et al.  Single and Cross-domain Polarity Classification using String Kernels , 2017, EACL.

[18]  Radu Tudor Ionescu,et al.  Learning to Identify Arabic and German Dialects using Multiple Kernels , 2017, VarDial.

[19]  Siu Cheung Hui,et al.  SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring , 2017, AAAI.

[20]  Aoife Cahill,et al.  String Kernels for Native Language Identification: Insights from Behind the Curtains , 2016, CL.

[21]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[22]  Cristian Grozea,et al.  Kernel Methods and String Kernels for Authorship Analysis , 2012, CLEF.

[23]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[24]  Helen Yannakoudakis,et al.  Automatic Text Scoring Using Neural Networks , 2016, ACL.

[25]  Yue Zhang,et al.  Automatic Features for Essay Scoring – An Empirical Study , 2016, EMNLP.

[26]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[27]  Nello Cristianini,et al.  Classification using String Kernels , 2000 .

[28]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[29]  Radu Tudor Ionescu,et al.  The Story of the Characters, the DNA and the Native Language , 2013, BEA@NAACL-HLT.

[30]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[31]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[32]  Lizhen Liu,et al.  Discourse Mode Identification in Essays , 2017, ACL.

[33]  Helen Yannakoudakis,et al.  A New Dataset and Method for Automatically Grading ESOL Texts , 2011, ACL.

[34]  Radu Tudor Ionescu A Fast Algorithm for Local Rank Distance: Application to Arabic Native Language Identification , 2015, ICONIP.

[35]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[36]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[38]  Yue Zhang,et al.  Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring , 2017, CoNLL.