Using Latent Semantic Analysis to Score Short Answer Constructed Responses: Automated Scoring of the Consequences Test

Automated scoring based on Latent Semantic Analysis (LSA) has been successfully used to score essays and constrained short answer responses. Scoring tests that capture open-ended, short answer responses poses some challenges for machine learning approaches. We used LSA techniques to score short answer responses to the Consequences Test, a measure of creativity and divergent thinking that encourages a wide range of potential responses. Analyses demonstrated that the LSA scores were highly correlated with conventional Consequence Test scores, reaching a correlation of .94 with human raters and were moderately correlated with performance criteria. This approach to scoring short answer constructed responses solves many practical problems including the time for humans to rate open-ended responses and the difficulty in achieving reliable scoring.

[1]  Steven E. Stemler,et al.  Using the theory of successful intelligence as a framework for developing assessments in AP physics , 2009 .

[2]  R. Sternberg.  Successful intelligence: A model for testing intelligence beyond IQ tests , 2015 .

[3]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[4]  Danielle S. McNamara,et al.  Handbook of latent semantic analysis , 2007 .

[5]  T. Landauer.  Automatic Essay Assessment , 2003 .

[6]  Mark D. Shermis,et al.  State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration , 2014 .

[7]  Lynn A. Streeter,et al.  Automating Expertise in Collaborative Learning Environments. , 2010 .

[8]  Michael D. Mumford,et al.  Domain-Based Scoring in Divergent-Thinking Tests: Validation Evidence in an Occupational Sample , 1998 .

[9]  Nicole B. Kersting,et al.  Automated Scoring of Teachers’ Open-Ended Responses to Video Prompts , 2014 .

[10]  Yigal Attali,et al.  Validity and Reliability of Automated Essay Scoring , 2013 .

[11]  Lillian Lee Scribes,et al.  Latent Semantic Indexing , 2007 .

[12]  Peter V. W. Hartmann.  Spearman's law of diminishing returns: A look at age differentiation. , 2006 .

[13]  Heinz Holling,et al.  Spearman's law of diminishing returns. A meta-analysis , 2017 .

[14]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[15]  Libby Gerard,et al.  Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles , 2014, Educational Measurement: Issues and Practice.

[16]  Thomas Hofmann,et al.  Unsupervised Learning by Probabilistic Latent Semantic Analysis , 2004, Machine Learning.

[17]  J. Mcardle,et al.  Comparative longitudinal structural analyses of the growth and decline of multiple intellectual abilities over the life span. , 2002, Developmental psychology.

[18]  Leticia H. Anaya,et al.  Comparing Latent Dirichlet Allocation and Latent Semantic Analysis as Classifiers , 2011 .

[19]  David Rosen,et al.  LDA v. LSA: A Comparison of Two Computational Text Analysis Tools for the Functional Categorization of Patents , 2016, ICCBR Workshops.

[20]  Klaus Zechner,et al.  Automated Essay Scoring: Writing Assessment and Instruction , 2010 .

[21]  Michael D. Mumford,et al.  Assessment of leader problem-solving capabilities. , 2000 .

[22]  Sonia Bergamaschi,et al.  Comparing LDA and LSA Topic Models for Content-Based Movie Recommendation Systems , 2014, WEBIST.