Measuring incremental changes in word knowledge: Experimental validation and implications for learning and assessment

The goal of this study was to test a new technique for assessing vocabulary development. This technique is based on an algorithm for scoring the accuracy of word definitions using a continuous scale (Collins-Thompson & Callan, 2007). In an experiment with adult learners, target words were presented in six different sentence contexts, and the number of informative versus misleading contexts was systematically manipulated. Participants generated a target definition after each sentence, and the definition-scoring algorithm was used to assess the degree of accuracy on each trial. We observed incremental improvements in definition accuracy across trials. Moreover, learning curves were sensitive to the proportion of misleading contexts, the use of spaced versus massed practice, and individual differences, demonstrating the utility of this procedure for capturing specific experimental effects on the trajectory of word learning. We discuss the implications of these results for measurement of meaning, vocabulary assessment, and instructional design.

[1]  Msl Swanborn,et al.  Impact of reading purpose on incidental word learning from context , 2002 .

[2]  William E. Nagy,et al.  Learning Word Meanings From Context During Normal Reading , 1987 .

[3]  K. Stanovich.  Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. , 1986 .

[4]  Edward J. Kameenui,et al.  Vocabulary instruction : research to practice , 2004 .

[5]  Charles A. Perfetti,et al.  ERP measures of partial semantic knowledge: Left temporal indices of skill differences and lexical quality , 2009, Biological Psychology.

[6]  Margaret G. McKeown,et al.  The Acquisition of Word Meaning from Context by Children of High and Low Ability. Outstanding Dissertation Monograph 1985. , 1985 .

[7]  Stephen D. Christman,et al.  Semantic Representations of Word Meanings by the Cerebral Hemispheres , 2002, Brain and Language.

[8]  J. Oakhill,et al.  Individual Differences in the Inference of Word Meanings From Context: The Influence of Reading Comprehension, Vocabulary Knowledge, and Memory Capacity. , 2004 .

[9]  Kevyn Collins-Thompson,et al.  Automatic and Human Scoring of Word Definition Responses , 2007, HLT-NAACL.

[10]  D. Gorfein.  On the consequences of meaning selection: Perspectives on resolving lexical ambiguity. , 2001 .

[11]  C. Perfetti,et al.  Lexical Quality in the Brain: ERP Evidence for Robust Word Learning From Context , 2010, Developmental neuropsychology.

[12]  R. Schmidt,et al.  New Conceptualizations of Practice: Common Principles in Three Paradigms Suggest New Concepts for Training , 1992 .

[13]  Marcia C. Linn,et al.  The Science of Learning and the Learning of Science Introducing Desirable Difficulties , 2006 .

[14]  Charles A. Perfetti,et al.  In vivo testing of learning and instructional principles: The design and implementation of school-based experimentation. , 2009 .

[15]  D. Langenberg.  Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction , 2000 .

[16]  D. McNamara.  Reading both high-coherence and low-coherence texts: effects of text sequence and prior knowledge. , 2001, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[17]  H. Walberg,et al.  Matthew Effects in Education , 1983 .

[18]  Henry L. Roediger,et al.  Disparate Effects of Repeated Testing: Reconciling Ballard's (1913) and Bartlett's (1932) Results , 1992 .

[19]  K. Koedinger,et al.  Exploring the Assistance Dilemma in Experiments with Cognitive Tutors , 2007 .

[20]  Marianne Elshout-Mohr,et al.  The acquisition of word meanings as a cognitive learning process , 1981 .

[21]  E. T. Possing,et al.  Neural Correlates of Lexical Access during Visual Word Recognition , 2003, Journal of Cognitive Neuroscience.

[22]  M. Vitevitch,et al.  The neighborhood characteristics of malapropisms. , 1996, Language and speech.

[23]  Andrew Y. Ng,et al.  Learning random walk models for inducing word dependency distributions , 2004, ICML.

[24]  Kevyn Collins-Thompson,et al.  Information retrieval for language tutoring: an overview of the REAP project , 2004, SIGIR '04.

[25]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[26]  Maxine Eskénazi,et al.  Automatic Question Generation for Vocabulary Assessment , 2005, HLT.

[27]  Margaret G. McKeown,et al.  Some effects of the nature and frequency of vocabulary instruction on the knowledge and use of words , 1985 .

[28]  Steven A. Stahl,et al.  The Effects of Vocabulary Instruction: A Model-Based Meta-Analysis , 1986 .

[29]  John R. Anderson,et al.  Practice and Forgetting Effects on Vocabulary Memory: An Activation-Based Model of the Spacing Effect , 2005, Cogn. Sci..

[30]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[31]  Arthur C. Graesser,et al.  Coh-Metrix: Analysis of text on cohesion and language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[32]  A. Feingold.  The Validity of the Information and Vocabulary Subtests of the WAIS for Predicting College Achievement , 1983 .

[33]  M. Daneman,et al.  Individual differences in comprehending and producing words in context , 1986 .

[34]  C. M. de Glopper,et al.  Deriving the meaning of unknown words from multiple contexts , 2001 .

[35]  Jeffrey D. Karpicke,et al.  Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[36]  Susan M. Watts,et al.  Contextual Analysis in Naturally Occurring Prose: Effects of Passage Length, Word Frequency, and Grade. , 1990 .

[37]  Charles A. Perfetti,et al.  The lexical basis of comprehension skill. , 2001 .

[38]  William J. Rapaport,et al.  In Defense of Contextual Vocabulary Acquisition , 2005, CONTEXT.

[39]  William J. Rapaport,et al.  In Defense of Contextual Vocabulary Acquisition: How to Do Things with Words in Context , 2005 .

[40]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[41]  Charles A. Perfetti,et al.  Morphology in Word Identification: A Word-Experience Model That Accounts for Morpheme Frequency Effects , 2003 .

[42]  C. Perfetti,et al.  The lexical quality hypothesis , 2002 .

[43]  William J. Rapaport,et al.  What Is the “Context” for Contextual Vocabulary Acquisition? , 2003 .

[44]  Francis T. Durso,et al.  Partial knowledge of word meanings , 1991 .

[45]  Steven A. Stahl.  How Words Are Learned Incrementally over Multiple Exposures. , 2003 .