Interactive Activation and Mutual Constraint Satisfaction in Perception and Cognition

In a seminal 1977 article, Rumelhart argued that perception required the simultaneous use of multiple sources of information, allowing perceivers to optimally interpret sensory information at many levels of representation in real time as information arrives. Building on Rumelhart's arguments, we present the Interactive Activation hypothesis-the idea that the mechanism used in perception and comprehension to achieve these feats exploits an interactive activation process implemented through the bidirectional propagation of activation among simple processing units. We then examine the interactive activation model of letter and word perception and the TRACE model of speech perception, as early attempts to explore this hypothesis, and review the experimental evidence relevant to their assumptions and predictions. We consider how well these models address the computational challenge posed by the problem of perception, and we consider how consistent they are with evidence from behavioral experiments. We examine empirical and theoretical controversies surrounding the idea of interactive processing, including a controversy that swirls around the relationship between interactive computation and optimal Bayesian inference. Some of the implementation details of early versions of interactive activation models caused deviation from optimality and from aspects of human performance data. More recent versions of these models, however, overcome these deficiencies. Among these is a model called the multinomial interactive activation model, which explicitly links interactive activation and Bayesian computations. We also review evidence from neurophysiological and neuroimaging studies supporting the view that interactive processing is a characteristic of the perceptual processing machinery in the brain. In sum, we argue that a computational analysis, as well as behavioral and neuroscience evidence, all support the Interactive Activation hypothesis. The evidence suggests that contemporary versions of models based on the idea of interactive activation continue to provide a basis for efforts to achieve a fuller understanding of the process of perception.

[1]  K. P. Unnikrishnan,et al.  A feedback model of visual attention , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[2]  Judea Pearl,et al.  Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach , 1982, AAAI.

[3]  James L. McClelland,et al.  Matching Exact Posterior Probabilities in the Multinomial Interactive Activation Model , 2010 .

[4]  R W Schvaneveldt,et al.  An activation--verification model for letter and word recognition: the word-superiority effect. , 1982, Psychological review.

[5]  W. Ganong Phonetic categorization in auditory word perception. , 1980, Journal of experimental psychology. Human perception and performance.

[6]  Phillip J. Holcomb Tracking the Time-course of Spoken Word Production with Event-Related Potentials , 2012 .

[7]  D. Swinney Lexical access during sentence comprehension: (Re)consideration of context effects , 1979 .

[8]  K. Gegenfurtner,et al.  Memory modulates color appearance , 2006, Nature Neuroscience.

[9]  S. Grossberg How does the brain build a cognitive code , 1988 .

[10]  Stephen G Lomber,et al.  Reciprocal Modulatory Influences between Tonotopic and Nontonotopic Cortical Fields in the Cat , 2010, The Journal of Neuroscience.

[11]  M. Pitt,et al.  Lexical activation (and other factors) can mediate compensation for coarticulation , 2003 .

[12]  D. Bavelier,et al.  Cross-modal plasticity: where and how? , 2002, Nature Reviews Neuroscience.

[13]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[14]  P. Smolensky,et al.  Neural and conceptual interpretation of PDP models , 1986 .

[15]  James L. McClelland Stochastic interactive processes and the effect of context on perception , 1991, Cognitive Psychology.

[16]  Katherine J. Midgley,et al.  When beef primes reef more than leaf: orthographic information affects phonological priming in spoken word recognition. , 2009, Psychophysiology.

[17]  James L. McClelland,et al.  An interactive Hebbian account of lexically guided tuning of speech perception , 2006, Psychonomic bulletin & review.

[18]  Mark S. Seidenberg,et al.  Modeling the Successes and Failures of Interventions for Disabled Readers , 2003 .

[19]  David Silbersweig,et al.  Functional neuroanatomy of verbal self-monitoring , 1996 .

[20]  Katherine A. DeLong,et al.  Probabilistic word pre-activation during language comprehension inferred from electrical brain activity , 2005, Nature Neuroscience.

[21]  D. Lewkowicz,et al.  The emergence of multisensory systems through perceptual narrowing , 2009, Trends in Cognitive Sciences.

[22]  D. Massaro Testing between the TRACE model and the fuzzy logical model of speech perception , 1989, Cognitive Psychology.

[23]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[24]  William D Marslen-Wilson,et al.  Processing interactions and lexical access during word recognition in continuous speech , 1978, Cognitive Psychology.

[25]  David E. Rumelhart,et al.  Toward an interactive model of reading. , 1994 .

[26]  J. Sawusch,et al.  Lexical neighborhood effects in phonetic processing. , 1997, Journal of experimental psychology. Human perception and performance.

[27]  Martin Kronbichler,et al.  The visual word form area and the frequency with which words are encountered: evidence from a parametric fMRI study , 2004, NeuroImage.

[28]  W. Pitts,et al.  How we know universals; the perception of auditory and visual forms. , 1947, The Bulletin of mathematical biophysics.

[29]  J. Grainger,et al.  Sequential Effects of Phonological Priming in Visual Word Recognition , 2005, Psychological science.

[30]  S. Grossberg A Theory of Visual Coding : Memory , and Development , 2010 .

[31]  Moshe Bar,et al.  The effects of priming on frontal-temporal , 2008 .

[32]  M. Tanenhaus,et al.  Continuous mapping from sound to meaning in spoken-language comprehension: immediate effects of verb-based thematic constraints. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[33]  James R. Booth,et al.  Differential effects of orthographic and phonological consistency in cortex for children with and without reading impairment , 2008, Neuropsychologia.

[34]  Anne Cutler,et al.  Phonological Abstraction in the Mental Lexicon , 2006, Cogn. Sci..

[35]  James L. McClelland Integrating probabilistic models of perception and interactive neural networks: a historical and tutorial review , 2013, Front. Psychol..

[36]  N. Logothetis,et al.  Multistable phenomena: changing views in perception , 1999, Trends in Cognitive Sciences.

[37]  R. M. Warren Perceptual Restoration of Missing Speech Sounds , 1970, Science.

[38]  Elizabeth Jefferies,et al.  Lexical and semantic binding in verbal short-term memory , 2006 .

[39]  D. Norris,et al.  Shortlist B: a Bayesian model of continuous speech recognition. , 2008, Psychological review.

[40]  G. Kane Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[41]  M. Tanenhaus,et al.  Actions and affordances in syntactic ambiguity resolution. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[42]  Jennifer K. Bizley,et al.  Visual influences on ferret auditory cortex , 2009, Hearing Research.

[43]  Holger Mitterer,et al.  Recalibrating Color Categories Using World Knowledge , 2008, Psychological science.

[44]  Karl J. Friston Hierarchical Models in the Brain , 2008, PLoS Comput. Biol..

[45]  G. Altmann,et al.  Incremental interpretation at verbs: restricting the domain of subsequent reference , 1999, Cognition.

[46]  James R. Booth,et al.  Children with reading difficulties show differences in brain regions associated with orthographic processing during spoken language processing , 2010, Brain Research.

[47]  Thomas Dean,et al.  A Computational Model of the Cerebral Cortex , 2005, AAAI.

[48]  Lee D. Erman,et al.  The Hearsay-I Speech Understanding System: An Example of the Recognition Process , 1973, IEEE Transactions on Computers.

[49]  L. Fadiga,et al.  The Motor Somatotopy of Speech Perception , 2009, Current Biology.

[50]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: Part 2. The contextual enhancement effect and some tests and extensions of the model. , 1982, Psychological review.

[51]  G. Dell,et al.  Lexical access in aphasic and nonaphasic speakers. , 1997, Psychological review.

[52]  R M Warren,et al.  Some age differences in auditory perception. , 1971, Bulletin of the New York Academy of Medicine.

[53]  C. Frith,et al.  Functional neuroanatomy of verbal self-monitoring , 1996, NeuroImage.

[54]  James L. McClelland,et al.  The TRACE model of speech perception , 1986, Cognitive Psychology.

[55]  Tali Bitan,et al.  Developmental changes in brain regions involved in phonological and orthographic processing during spoken language processing , 2008, NeuroImage.

[56]  James L. McClelland,et al.  The Morton-Massaro law of information integration: implications for models of perception. , 2001, Psychological review.

[57]  Naomi H. Feldman,et al.  The influence of categories on perception: explaining the perceptual magnet effect as optimal statistical inference. , 2009, Psychological review.

[58]  J. Ashby,et al.  Prosodic phonological representations early in visual word recognition. , 2008, Journal of experimental psychology. Human perception and performance.

[59]  James S. Magnuson,et al.  Lexical effects on compensation for coarticulation: the ghost of Christmash past , 2003, Cogn. Sci..

[60]  B. Rapp,et al.  Discreteness and interactivity in spoken word production. , 2000, Psychological review.

[61]  W. Singer,et al.  Activation of Heschl’s Gyrus during Auditory Hallucinations , 1999, Neuron.

[62]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[63]  Karl J. Friston,et al.  Different activation patterns in the visual cortex of late and congenitally blind subjects. , 1998, Brain : a journal of neurology.

[64]  Emily B. Myers,et al.  The neural bases of the lexical effect: an fMRI investigation. , 2008, Cerebral cortex.

[65]  Tobias Teichert,et al.  The dorsal medial frontal cortex is sensitive to time on task, not response conflict or error likelihood , 2011, NeuroImage.

[66]  M. Sigman,et al.  Opinion TRENDS in Cognitive Sciences Vol.9 No.7 July 2005 The neural code for written words: a proposal , 2022 .

[67]  James L. McClelland,et al.  Generalization Through the Recurrent Interaction of Episodic Memories , 2012, Psychological review.

[68]  Régine Kolinsky,et al.  The orthographic consistency effect in the recognition of French spoken words: An early developmental shift from sublexical to lexical orthographic activation , 2009 .

[69]  M. Pitt The locus of the lexical shift in phoneme identification. , 1995, Journal of experimental psychology. Learning, memory, and cognition.

[70]  Marie Bienkowski,et al.  Automatic access of the meanings of ambiguous words in context: Some limitations of knowledge-based processing , 1982, Cognitive Psychology.

[71]  J. A. Pruszynski,et al.  Neural correlates , 2023 .

[72]  A. Samuel,et al.  Perceptual learning for speech , 2009, Attention, perception & psychophysics.

[73]  D. Massaro,et al.  The role of lateral masking and orthographic structure in letter and word recognition. , 1979, Acta psychologica.

[74]  Lori L. Holt,et al.  Effects of Attention on the Strength of Lexical Influences on Speech Perception: Behavioral Experiments and Computational Mechanisms , 2008, Cogn. Sci..

[75]  M. Bar Visual objects in context , 2004, Nature Reviews Neuroscience.

[76]  William W. Graves,et al.  Neural Systems for Reading Aloud: A Multiparametric Approach , 2009, Cerebral cortex.

[77]  M. Pitt,et al.  Is Compensation for Coarticulation Mediated by the Lexicon , 1998 .

[78]  Heike Martensen,et al.  The lexical bias effect is modulated by context, but the standard monitoring account doesn’t fly: Related beply to Baars et al. (1975) ☆ , 2005 .

[79]  E Donchin,et al.  Brain potentials as indices of orthographic and phonological interaction during word matching. , 1987, Journal of experimental psychology. Learning, memory, and cognition.

[80]  D. Norris,et al.  Perceptual learning in speech , 2003, Cognitive Psychology.

[81]  G. Miller,et al.  Cognitive science. , 1981, Science.

[82]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[83]  Carson C. Chow,et al.  Repetition priming and repetition suppression: A case for enhanced efficiency through neural synchronization , 2012, Cognitive neuroscience.

[84]  Apostolos P. Georgopoulos,et al.  Cognitive dimensions of orthographic stimuli affect occipitotemporal dynamics , 2005, Experimental Brain Research.

[85]  D. Norris,et al.  No lexical–prelexical feedback during speech perception or: Is it time to stop playing those Christmas tapes? , 2009 .

[86]  Jeffrey R. Binder,et al.  Tuning of the human left fusiform gyrus to sublexical orthographic structure , 2006, NeuroImage.

[87]  D. Massaro,et al.  Integration versus interactive activation: The joint influence of stimulus and context in perception , 1991, Cognitive Psychology.

[88]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[89]  Arthur G. Samuel,et al.  Early levels of analysis of speech. , 1996 .

[90]  A. Samuel Lexical Activation Produces Potent Phonemic Percepts , 1997, Cognitive Psychology.

[91]  R. Aslin,et al.  Anticipatory Eye Movements Reveal Infants' Auditory and Visual Categories. , 2004, Infancy : the official journal of the International Society on Infant Studies.

[92]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[93]  Michael J. Spivey,et al.  Syntactic ambiguity resolution in discourse: modeling the effects of referential context and lexical frequency. , 1998, Journal of experimental psychology. Learning, memory, and cognition.

[94]  Deb Roy,et al.  Semantic context effects on color categorization , 2009 .

[95]  Geoffrey E. Hinton,et al.  The appeal of parallel distributed processing , 1986 .

[96]  D W Massaro,et al.  Letter information and orthographic context in word perception. , 1979, Journal of experimental psychology. Human perception and performance.

[97]  G. Rizzolatti,et al.  Reorienting attention across the horizontal and vertical meridians: Evidence in favor of a premotor theory of attention , 1987, Neuropsychologia.

[98]  Geoffrey E. Hinton Where Do Features Come From? , 2014, Cogn. Sci..

[99]  Peter Hagoort,et al.  Electrophysiological evidence for early contextual influences during spoken-word recognition: The N2000 , 2000 .

[100]  P. Kuhl Human adults and human infants show a “perceptual magnet effect” for the prototypes of speech categories, monkeys do not , 1991, Perception & psychophysics.

[101]  Caden Salvata,et al.  Temporal Cortex Reflects Effects of Sentence Context on Phonetic Processing , 2013, Journal of Cognitive Neuroscience.

[102]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[103]  Jean Vroomen,et al.  Recalibration of phonetic categories by lipread speech versus lexical information. , 2007, Journal of experimental psychology. Human perception and performance.

[104]  Refractor Vision , 2000, The Lancet.

[105]  A. Samuel Phonemic restoration: insights from a new methodology. , 1981, Journal of experimental psychology. General.

[106]  Joseph D. W. Stephens,et al.  Preceding phonetic context affects perception of nonspeech (L) , 2003 .

[107]  James L. McClelland,et al.  A PDP model of the simultaneous perception of multiple objects , 2011, Connect. Sci..

[108]  C. Eriksen,et al.  Pre- and poststimulus activation of response channels: a psychophysiological analysis. , 1988, Journal of experimental psychology. Human perception and performance.

[109]  James L. McClelland,et al.  The role of familiar units in perception of words and nonwords , 1977 .

[110]  T. S. Lee,et al.  Dynamics of subjective contour formation in the early visual cortex. , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[111]  James L. McClelland The Case for Interactionism in Language Processing. , 1987 .

[112]  Bruce D. McCandliss,et al.  The visual word form area: expertise for reading in the fusiform gyrus , 2003, Trends in Cognitive Sciences.

[113]  D. Plaut,et al.  Locating Object Knowledge in the Brain , 2022 .

[114]  D Norris,et al.  Merging information in speech recognition: Feedback is never necessary , 2000, Behavioral and Brain Sciences.

[115]  Seppo P. Ahlfors,et al.  Lexical influences on speech perception: A Granger causality analysis of MEG and EEG source estimates , 2008, NeuroImage.

[116]  Martin Kronbichler,et al.  Taxi vs. Taksi: On Orthographic Word Recognition in the Left Ventral Occipitotemporal Cortex , 2007, Journal of Cognitive Neuroscience.

[117]  A G Samuel,et al.  Knowing a Word Affects the Fundamental Perception of The Sounds Within it , 2001, Psychological science.

[118]  P. Barone,et al.  Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey , 2005, The European journal of neuroscience.

[119]  G. M. Reicher Perceptual recognition as a function of meaninfulness of stimulus material. , 1969, Journal of experimental psychology.

[120]  D. Massaro,et al.  Phonological context in speech perception , 1983, Perception & psychophysics.

[121]  S. Monsell,et al.  Lexical and sublexical translation of spelling to sound : Strategic anticipation of lexical status , 1992 .

[122]  G S Dell,et al.  A spreading-activation theory of retrieval in sentence production. , 1986, Psychological review.

[123]  D. Rumelhart,et al.  Process of recognizing tachistoscopically presented words. , 1974, Psychological review.

[124]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[125]  David A. Medler,et al.  Neural correlates of sensory and decision processes in auditory object identification , 2004, Nature Neuroscience.

[126]  D. Heeger Normalization of cell responses in cat striate cortex , 1992, Visual Neuroscience.

[127]  Kathleen S Rockland,et al.  Multisensory convergence in calcarine visual areas in macaque monkey. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[128]  M. Carreiras,et al.  From numbers to letters: Feedback regularization in visual word recognition , 2010, Neuropsychologia.

[129]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[130]  J. M. Hupé,et al.  Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons , 1998, Nature.

[131]  Jeffrey L. Elman,et al.  Speech Perception as a Cognitive Process: The Interactive Activation Model. , 1983 .

[132]  James S. Magnuson,et al.  Immediate effects of form-class constraints on spoken word recognition , 2008, Cognition.

[133]  R. H. Phaf,et al.  SLAM: A connectionist model for attention in visual selection tasks , 1990, Cognitive Psychology.

[134]  E. Marshall,et al.  NIMH: caught in the line of fire without a general , 1995, Science.

[135]  A. Samuel Red herring detectors and speech perception: In defense of selective adaptation , 1986, Cognitive Psychology.

[136]  Stéphane Dufau,et al.  On-line Orthographic Influences on Spoken Language in a Semantic Task , 2009, Journal of Cognitive Neuroscience.

[137]  Colin M. Brown,et al.  Electrophysiological Evidence for Early Contextual Influences during Spoken-Word Recognition: N200 Versus N400 Effects , 2001, Journal of Cognitive Neuroscience.

[138]  J. Hirsch,et al.  A Neural Representation of Categorization Uncertainty in the Human Brain , 2006, Neuron.

[139]  D. Norris Models of visual word recognition , 2013, Trends in Cognitive Sciences.

[140]  Jeffrey S Bowers,et al.  On the biological plausibility of grandmother cells: implications for neural network theories in psychology and neuroscience. , 2009, Psychological review.

[141]  C. Schroeder,et al.  Somatosensory input to auditory association cortex in the macaque monkey. , 2001, Journal of neurophysiology.

[142]  Robert Allen Fox,et al.  Effect of Lexical Status on Phonetic Categorization , 1984 .

[143]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[144]  James L. McClelland,et al.  Cognitive penetration of the mechanisms of perception: Compensation for coarticulation of lexically restored phonemes , 1988 .

[145]  T. Powell,et al.  An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. , 1970, Brain : a journal of neurology.

[146]  David Mumford,et al.  On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[147]  Stephen G. Lomber,et al.  Auditory cortex projections target the peripheral field representation of primary visual cortex , 2008, Experimental Brain Research.

[148]  D Mumford,et al.  On the computational architecture of the neocortex. II. The role of cortico-cortical loops. , 1992, Biological cybernetics.

[149]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[150]  Giuseppe Iaria,et al.  Encoding in the Visual Word Form Area: An fMRI Adaptation Study of Words versus Handwriting , 2010, Journal of Cognitive Neuroscience.

[151]  M. Hallett,et al.  Functional relevance of cross-modal plasticity in blind humans , 1997, Nature.

[152]  A. Borst Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[153]  James L. McClelland The programmable blackboard model of reading , 1986 .

[154]  B. Bergum,et al.  Attention and Performance VI , 1978 .

[155]  Nadia E. Cone,et al.  Neural correlates of orthographic and phonological consistency effects in children , 2008, Human brain mapping.

[156]  James L. McClelland Putting Knowledge in its Place: A Scheme for Programming Parallel Processing Structures on the Fly , 1988, Cogn. Sci..

[157]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[158]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[159]  S. Grossberg How does a brain build a cognitive code , 1980 .

[160]  Charles A. Perfetti,et al.  Interactive Processes in Reading , 1981 .

[161]  James L. McClelland,et al.  A parallel distributed processing approach to automaticity. , 1992, The American journal of psychology.

[162]  Thomas L. Griffiths,et al.  One and Done? Optimal Decisions From Very Few Samples , 2014, Cogn. Sci..

[163]  Matthew H. Davis,et al.  Predictive Top-Down Integration of Prior Knowledge during Speech Perception , 2012, The Journal of Neuroscience.

[164]  Johannes C. Ziegler,et al.  On-line activation of orthography in spoken word recognition , 2008, Brain Research.

[165]  H. F. J. M. Buffart,et al.  Formal theories of visual perception , 1978 .

[166]  V. Mann,et al.  Influence of preceding fricative on stop consonant perception. , 1981, The Journal of the Acoustical Society of America.

[167]  Raj Reddy,et al.  The HEARSAY Speech Understanding System , 1974 .

[168]  Karl J. Friston Learning and inference in the brain , 2003, Neural Networks.

[169]  James L. McClelland,et al.  A distributed, developmental model of word recognition and naming. , 1989, Psychological review.

[170]  Jyrki Tuomainen,et al.  Lexical effects on auditory speech perception: An electrophysiological study , 2007, Neuroscience Letters.

[171]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[172]  James L. McClelland,et al.  Locating object knowledge in the brain: comment on Bowers's (2009) attempt to revive the grandmother cell hypothesis. , 2010, Psychological review.