Wisdom of Students: A Consistent Automatic Short Answer Grading Technique

Automatic short answer grading (ASAG) techniques are designed to automatically assess short answers written in natural language having a length of a few words to a few sentences. In this paper, we report an intriguing finding that the set of short answers to a question, collectively, share significant lexical commonalities. Based on this finding, we propose an unsupervised ASAG technique that only requires sequential pattern mining in the first step and an intuitive scoring process in the second step. We demonstrate, using multiple datasets, that the proposed technique effectively exploits wisdom of students to deliver comparable or better performance than prior ASAG techniques as well as distributional semantics-based approaches that require heavy training with a large corpus. Moreover, by virtue of being independent of instructor provided model answers, our technique offers consistency by overcoming the limitation of undesired variability in performance exhibited by existing unsupervised techniques.

[1]  Wael Hassan Gomaa,et al.  Short Answer Grading Using String Similarity And Corpus-Based Similarity , 2012 .

[2]  Enrique Alfonseca,et al.  Application of the BLEU Method for Evaluating Free-text Answers in an E-learning Environment , 2004, LREC.

[3]  Ramakrishnan Srikant,et al.  Mining sequential patterns , 1995, Proceedings of the Eleventh International Conference on Data Engineering.

[4]  Angelo Kyrilov,et al.  Automated assessment of short free-text responses in computer science using latent semantic analysis , 2011, ITiCSE '11.

[5]  Tom Mitchell,et al.  Towards robust computerised marking of free-text responses , 2002 .

[6]  Randy M. Kaplan,et al.  Using Lexical Semantic Techniques to Classify Free-Responses , 1999 .

[7]  Enrique Alfonseca,et al.  Automatic Assessment of Open Ended Questions with a Bleu-Inspired Algorithm and Shallow NLP , 2004, EsTAL.

[8]  Christian Gütl.  Moving towards a Fully Automatic Knowledge Assessment Tool , 2008, iJET.

[9]  Arthur C. Graesser,et al.  Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor , 2000, Interact. Learn. Environ..

[10]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[11]  Chun-Yen Chang,et al.  Assessing Creative Problem-solving with Automated Text Grading , 2008 .

[12]  Nitin Madnani,et al.  Automated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension , 2013, BEA@NAACL-HLT.

[13]  Peter Wiemer-Hastings,et al.  Rules for Syntax, Vectors for Semantics , 2001 .

[14]  Diana Pérez-Marín,et al.  Willow: a system to automatically assess students’ free-text answers by using a combination of shallow NLP techniques , 2011 .

[15]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[16]  Jana Sukkarieh.  Using a MaxEnt Classifier for the Automatic Content Scoring of Free‐Text Responses , 2011 .

[17]  Zellig S. Harris,et al.  Mathematical structures of language , 1968, Interscience tracts in pure and applied mathematics.

[18]  Victoria Johansson,et al.  Lexical diversity and lexical density in speech and writing , 2009 .

[19]  Chris Brew,et al.  SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge , 2013, *SEMEVAL.

[20]  Carlo Strapparava,et al.  Automatic Assessment of Students' Free-Text Answers Underpinned by the Combination of a BLEU-Inspired Algorithm and Latent Semantic Analysis , 2005, FLAIRS Conference.

[21]  Evgeniy Gabrilovich,et al.  Overcoming the Brittleness Bottleneck using Wikipedia: Enhancing Text Categorization with Encyclopedic Knowledge , 2006, AAAI.

[22]  Rada Mihalcea,et al.  Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments , 2011, ACL.

[23]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[24]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[25]  Surendra Prasad,et al.  Automatic Evaluation of Students’ Answers using Syntactically Enhanced LSA , 2003, HLT-NAACL 2003.

[26]  Yasuyo Sawaki,et al.  A Reliable Approach to Automatic Assessment of Short Answer Free Responses , 2002, COLING.

[27]  Shourya Roy,et al.  A Perspective on Computer Assisted Assessment Techniques for Short Free-Text Answers , 2015, CAA.

[28]  Peter W. Foltz,et al.  Identifying Patterns For Short Answer Scoring Using Graph-based Lexico-Semantic Text Matching , 2015, BEA@NAACL-HLT.

[29]  Rada Mihalcea,et al.  Text-to-Text Semantic Similarity for Automatic Short Answer Grading , 2009, EACL.

[30]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[31]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[32]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[33]  David W. Conrath,et al.  Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy , 1997, ROCLING/IJCLCLP.

[34]  Benno Stein,et al.  The Eras and Trends of Automatic Short Answer Grading , 2015, International Journal of Artificial Intelligence in Education.

[35]  Anne Laurent,et al.  Sequential patterns for text categorization , 2006, Intell. Data Anal..