The Use of Latent Semantic Analysis as a Tool for the Quantitative Assessment of Understanding and Knowledge

Latent Semantic Analysis (LSA) is a statistical model of word usage that has been used for a variety of applications. One of these applications is the quantitative assessment of the semantic content within written text. While the technology has been successful in correlating with the qualitative ratings of human experts, it is unclear what aspect of knowledge is being reflected in an LSA output. The two experiments presented here were designed to address this general question. We were particularly interested in whether an LSA analysis more accurately reflects the factual or conceptual knowledge contained in written material. Experiment 1 explored this issue by comparing LSA analyses of essays to human-generated scores. It also compared the LSA output to several measures of conceptual structure. Experiment 2 correlated LSA analyses of transcribed recall protocols with a series of comprehension measures that were designed to vary in the degree to which they reflect conceptual or factual knowledge. We found compelling evidence that LSA analyses are a stronger reflection of the text-based knowledge represented by essays and recall protocols than conceptual knowledge. Both studies also explored a methodological issue pertaining to the use of LSA. Specifically, does LSA have to be “trained” in the particular content area of the text to be analyzed? This question was addressed by running multiple LSA analyses, each performed with differing “semantic spaces” created through training in domain specific or general content areas. We found that LSA performed best when trained in a content area specific to the material to be analyzed. These results are discussed with respect to the application of LSA analyses in the classroom and laboratory.

[1]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[2]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[3]  W. Kintsch,et al.  Are Good Texts Always Better? Interactions of Text Coherence, Background Knowledge, and Levels of Understanding in Learning From Text , 1996 .

[4]  Moshe Naveh-Benjamin,et al.  Use of the Ordered-Tree Technique to Assess Students' Initial Knowledge and Conceptual Learning , 1989 .

[5]  Susan R. Goldman,et al.  Students making sense of informational text: Relations between processing and representation , 1998 .

[6]  Judith S Reitman,et al.  Organization revealed by recall orders and confirmed by pauses , 1980, Cognitive Psychology.

[7]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[8]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[9]  Bob Rehder,et al.  Using latent semantic analysis to assess knowledge: Some technical considerations , 1998 .

[10]  Michael Green,et al.  Assessing the Impact of a Proposed Expert System via Simulation , 1995 .

[11]  B. K. Britton,et al.  Using Kintsch's computational model to improve instructional text: Effects of repairing inference calls on recall and cognitive structures. , 1991 .

[12]  W. Kintsch,et al.  Learning from text: Structural knowledge assessment in the study of discourse comprehension , 1999 .

[13]  W. C. Robertson,et al.  Detection of Cognitive Structure with Protocol Data: Predicting Performance on Physics Transfer Problems , 1990, Cogn. Sci..

[14]  M. Chi,et al.  Network representation of a child's dinosaur knowledge. , 1983 .

[15]  Peter W. Foltz,et al.  Reasoning from Multiple Texts: An Automatic Analysis of Readers? Situation Models , 1996 .

[16]  Amy M. Shapiro,et al.  The Relationship between Prior Knowledge and Interactive Overviews during Hypermedia-Aided Learning , 1999 .

[17]  Danielle S. McNamara,et al.  Learning from texts: Effects of prior knowledge and text coherence , 1996 .