Assessing Reading Skill With a Think-Aloud Procedure and Latent Semantic Analysis

The viability of assessing reading strategies is studied based on think-aloud protocols combined with Latent Semantic Analysis (LSA). Readers in two studies thought aloud after reading specific focal sentences embedded in two stories. LSA was used to estimate the semantic similarity between readers' think-aloud protocols to the focal sentences and sentences in the stories that provided direct causal antecedents to the focal sentences. Study 1 demonstrated that according to human- and LSA-based assessments of the protocols, the responses of less-skilled readers semantically overlapped more with the focal sentences than with the causal antecedent sentences, whereas the responses of skilled readers overlapped with these sentences equally. In addition, the extent that the semantic overlap with causal antecedents was greater than the overlap with the focal sentences predicted performance on comprehension test questions and the Nelson-Denny test of reading skill. Study 2 replicated these findings and also demonstrated that the semantic overlap scores (based on the protocols) predicted recall for stories that were read silently. Together, the findings supported the viability of developing a computerized assessment tool using verbal protocols and LSA.

[1]  Arthur C. Graesser,et al.  Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor , 2000, Interact. Learn. Environ..

[2]  Douglas J. Hacker,et al.  Metacognition in educational theory and practice. , 1998 .

[3]  C. Steele.  A threat in the air. How stereotypes shape intellectual identity and performance. , 1997, The American psychologist.

[4]  Rolf A. Zwaan,et al.  The influence of language proficiency and comprehension skill on situation‐model construction , 1996 .

[5]  Robert F. Lorch,et al.  Network representations of causal relations in memory for narrative texts: Evidence from primed recognition , 1993 .

[6]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994, Psychological review.

[7]  Joseph P. Magliano,et al.  Conscious understanding during comprehension , 1996 .

[8]  T. Trabasso,et al.  Understanding text: Achieving explanatory coherence through on‐line inferences and mental operations in working memory , 1993 .

[9]  Hyun-Jeong Joyce Kim,et al.  Updating Situation Models From Descriptive Texts: A Test of the Situational Operator Model , 2000 .

[10]  C. Perfetti.  Psycholinguistics and reading ability. , 1994 .

[11]  Jane Oakhill,et al.  Individual differences in children's text comprehension. , 1994 .

[12]  C. Steele,et al.  Stereotype threat and the intellectual test performance of African Americans. , 1995, Journal of personality and social psychology.

[13]  G. Radvansky,et al.  The Fate of Completed Goal Information in Narrative Comprehension , 1997 .

[14]  Peter Afflerbach,et al.  Verbal Protocols of Reading: The Nature of Constructively Responsive Reading , 1996 .

[15]  Paul Whitney,et al.  Working‐memory capacity and the use of elaborative inferences in text comprehension , 1991 .

[16]  Gerald S. Hanna,et al.  Studies of the Seriousness of Three Threats to Passage Dependence , 1980 .

[17]  Claude M. Steele,et al.  Stereotyping and its threat are real. , 1998 .

[18]  Annemarie Sullivan Palincsar,et al.  Interactive Teaching to Promote Independent Learning from Text. , 1986 .

[19]  M. Daneman,et al.  A new tool for measuring and understanding individual differences in the component processes of reading comprehension , 2001 .

[20]  Michelene T. H. Chi,et al.  Self-Explanations: How Students Study and Use Examples in Learning To Solve Problems. Technical Report No. 9. , 1987 .

[21]  R. Freedle,et al.  Can Multiple-Choice Reading Tests Be Construct-Valid? A Reply to Katz, Lautenschlager, Blackburn, and Harris , 1994 .

[22]  Katja Wiemer-Hastings,et al.  Using latent semantic analysis to assess reader strategies , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[23]  P. Broek.  Comprehension and memory of narrative texts: Inferences and coherence. , 1994 .

[24]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[25]  Edward M. Hundert,et al.  The Cement of the Universe , 1990 .

[26]  M. Gernsbacher,et al.  Investigating differences in general comprehension skill. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[27]  T. Trabasso,et al.  Inferences during Reading: Converging Evidence from Discourse Analysis, Talk-Aloud Protocols, and Recognition Priming , 1993 .

[28]  Huub van den Bergh,et al.  On the Construct Validity of Multiple- Choice Items for Reading Comprehension , 1990 .

[29]  Susan R. Goldman,et al.  Memory for embedded and sequential story structures , 1986 .

[30]  T. Trabasso,et al.  Causal thinking and the representation of narrative events , 1985 .

[31]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[32]  Arthur C. Graesser,et al.  Strategic processing during comprehension. , 1999 .

[33]  T. Trabasso,et al.  Causal relatedness and importance of story events , 1985 .

[34]  Ronald P. Carver,et al.  What Do Standardized Tests of Reading Comprehension Measure in Terms of Efficiency, Accuracy, and Rate?. , 1992 .

[35]  T. Trabasso,et al.  Logical necessity and transitivity of causal relations in stories , 1989 .

[36]  Morton Ann Gernsbacher,et al.  Less Skilled Readers Have Less Efficient Suppression Mechanisms , 1993, Psychological science.

[37]  J. B. Wyman,et al.  What is reading ability , 1921 .

[38]  Morton Ann Gernsbacher,et al.  Two Decades of Structure Building. , 1997, Discourse processes.

[39]  Robert Pritchard,et al.  A Description of What Happens When an Examinee Takes a Multiple-Choice Reading Comprehension Test , 1990 .

[40]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[41]  T. Trabasso,et al.  Modeling causal integration and availability of information during comprehension of narrative texts. , 1999 .

[42]  Stuart Katz,et al.  Answering Reading Comprehension Items without the Passages on the SAT–I , 1999 .

[43]  Debra L. Long,et al.  The strategic nature of less skilled readers’ suppression problems , 1999 .

[44]  J. Earman,et al.  The Cement Of The Universe , 1974 .

[45]  Bob Rehder,et al.  How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans , 1997 .

[46]  Matthew W. Lewis,et al.  Self-Explonations: How Students Study and Use Examples in Learning to Solve Problems , 1989, Cogn. Sci..

[47]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[48]  J. Glover,et al.  Utility of the Nelson-Denny as a Predictor of Structure and Thematicity in Memory from Prose , 1979 .