Parameters driving effectiveness of automated essay scoring with LSA

Automated essay scoring with latent semantic analysis (LSA) has recently been subject to increasing interest. Although previous authors have achieved grade ranges similar to those awarded by humans, it is still not clear which and how parameters improve or decrease the effectiveness of LSA. This pa-per presents an analysis of the effects of these parameters, such as text pre-processing, weighting, singular value dimensionality and type of similarity measure, and benchmarks this effectiveness by comparing machine-assigned with human-assigned scores in a real-world case. We show that each of the identified factors significantly influences the quality of automated essay scor-ing and that the factors are not independent of each other.

[1]  D. Whittington,et al.  Approaches to the computerized assessment of free text responses , 1999 .

[2]  Susan T. Dumais,et al.  Using Linear Algebra for Intelligent Information Retrieval , 1995, SIAM Rev..

[3]  Charles A. Perfetti,et al.  The limits of co‐occurrence: Tools and theories in language research , 1998 .

[4]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[5]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[6]  Anthony G. Picciano.  Educational research primer , 2004 .

[7]  Preslav Nakov,et al.  Weight functions impact on LSA performance , 2001 .

[8]  Preslav Nakov,et al.  Towards Deeper Understanding of the LSA Performance , 2003 .

[9]  Peter W. Foltz,et al.  The Debate on Automated Essay Grading , 2000, IEEE Intell. Syst..

[10]  Thomas K. Landauer,et al.  Simulating Text Understanding for Educational Applications with Latent Semantic Analysis: Introduction to LSA , 2000, Interact. Learn. Environ..

[11]  Arthur C. Graesser,et al.  AutoTutor: A simulation of a human tutor , 1999, Cognitive Systems Research.

[12]  William Wresch,et al.  The Imminence of Grading Essays by Computer-25 Years Later , 1993 .

[13]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[14]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[15]  Susan T. Dumais,et al.  Enhancing Performance in Latent Semantic Indexing (LSI) Retrieval , 1990 .