The Application of LSA to the Evaluation of Questionnaire Responses

This chapter will discuss research surrounding the development of applications for automated evaluation and generation of instructional feedback based on the analysis of responses to open-ended essay questions. While an effective means of evaluation, examining responses to essay questions requires natural language processing that often demands human input and guidance which can be labor intensive, time consuming, and language dependent. In order to provide this means of evaluation on a larger scale, an automated unsupervised learning approach is necessary that overcomes all these limitations. Latent Semantic Analysis (LSA) is an unsupervised learning system used for deriving and representing the semantic relationships between items in a body of natural language content. It mimics the representation of meaning that is formed by a human who learns linguistic constructs through exposure to natural language over time. The applications described in this chapter leverage LSA as an unsupervised system to learn language and provide a semantic framework that can be used for mapping natural language responses, evaluating the quality of those responses, and identifying relevant instructional feedback based on their semantic content. We will discuss the learning algorithms used to construct the LSA framework of meaning as well as methods to apply that framework for the evaluation and generation of feedback.

[1]  Thomas K. Landauer,et al.  An LSA-Based Software Tool for Matching Jobs, People, and Instruction , 2000, Interact. Learn. Environ..

[2]  Marti A. Hearst.  The debate on automated essay grading , 2000 .

[3]  T. Landauer.  Automatic Essay Assessment , 2003 .

[4]  Lynn A. Streeter,et al.  Automating Expertise in Collaborative Learning Environments. , 2010 .

[5]  William Wresch,et al.  The Imminence of Grading Essays by Computer-25 Years Later , 1993 .

[6]  Arthur C. Graesser,et al.  Automated Evaluation of Text and Discourse with Coh-Metrix: Coh-Metrix Measures , 2014 .

[7]  Arthur C. Graesser,et al.  Using LSA in AutoTutor: Learning through mixed-initiative dialogue in natural language. , 2007 .

[8]  Susan T. Dumais,et al.  The Vocabulary Problem in Human-System Communication: an Analysis and a Solution , 1987 .

[9]  Michael W. Berry,et al.  Latent Semantic Indexing , 2010 .

[10]  Peter W. Foltz,et al.  Learning Human-like Knowledge by Singular Value Decomposition: A Progress Report , 1997, NIPS.

[11]  Karen E. Lochbaum,et al.  A New Formative Assessment Technology for Reading and Writing , 2009 .

[12]  Arthur C. Graesser,et al.  AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back , 2012, TIIS.

[13]  Peter W. Foltz.  Automated content processing of spoken and written discourse: Text coherence, essays, and team analyses , 2005 .

[14]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[15]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[16]  T. Landauer.  LSA as a Theory of Meaning , 2007 .

[17]  Peter W. Foltz,et al.  Learning from text: Matching readers and texts by latent semantic analysis , 1998 .

[18]  Lynn A. Streeter,et al.  Automated Tools for Collaborative Learning Environments , 2007 .

[19]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[20]  Bob Rehder,et al.  Using latent semantic analysis to assess knowledge: Some technical considerations , 1998 .

[21]  T. Landauer.  Learning and Representing Verbal Meaning , 1998 .

[22]  Peter W. Foltz,et al.  Supporting Content-Based Feedback in On-Line Writing Evaluation with LSA , 2000, Interact. Learn. Environ..

[23]  Yiming Yang,et al.  RCV1: A New Benchmark Collection for Text Categorization Research , 2004, J. Mach. Learn. Res..

[24]  Peter W. Foltz,et al.  Models of Vocabulary Acquisition: Direct Tests and Text-Derived Simulations of Vocabulary Growth , 2014 .

[25]  Michael W. Berry,et al.  Mathematical Foundations Behind Latent Semantic Analysis , 2007 .

[26]  E. B. Page.  Computer Grading of Student Prose, Using Modern Concepts and Software , 1994 .

[27]  Peter W. Foltz,et al.  Latent semantic analysis for text-based research , 1996 .

[28]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[29]  Michael W. Berry,et al.  Principal Component Analysis for Information Retrieval , 2005 .

[30]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[31]  Arthur C. Graesser,et al.  Automated Evaluation of Text and Discourse with Coh-Metrix: List of Tables , 2014 .

[32]  Thomas K. Landauer,et al.  On the computational basis of learning and cognition: Arguments from LSA , 2002 .

[33]  Lynn A. Streeter,et al.  Technology as a Tool for Leadership Development: Effectiveness of Automated Web-Based Systems in Facilitating Tacit Knowledge Acquisition , 2008 .

[34]  E. B. Page,et al.  The Computer Moves into Essay Grading: Updating the Ancient Test. , 1995 .

[35]  Michael W. Berry,et al.  Out-of-core SVD performance for document indexing , 2007 .

[36]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[37]  S. T. Dumais,et al.  Using latent semantic analysis to improve access to textual information , 1988, CHI '88.

[38]  Susan T. Dumais,et al.  Improving the retrieval of information from external sources , 1991 .

[39]  Shailesh S. Kulkarni,et al.  The Use of Latent Semantic Analysis in Operations Management Research , 2014, Decis. Sci..

[40]  P. Foltz,et al.  Application of latent semantic analysis for open-ended responses in a large, epidemiologic study , 2011, BMC medical research methodology.