Evidence for a deep, distributed and dynamic semantic code in human ventral anterior temporal cortex

How does the human brain encode semantic information about objects? This paper reconciles two seemingly contradictory views. The first proposes that local neural populations independently encode semantic features; the second, that semantic representations arise as a dynamic distributed code that changes radically with stimulus processing. Combining simulations with a well-known neural network model of semantic memory, multivariate pattern classification, and human electrocorticography, we find that both views are partially correct: semantic information is distributed across ventral temporal cortex in a dynamic code that possesses stable feature-like elements in posterior regions but with elements that change rapidly and nonlinearly in anterior regions. This pattern is consistent with the view that anterior temporal lobes serve as a deep cross-modal “hub” in an interactive semantic network, and more generally suggests that tertiary association cortices may adopt dynamic distributed codes difficult to detect with common brain imaging methods.

[1]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[2]  James L. McClelland,et al.  The parallel distributed processing approach to semantic cognition , 2003, Nature Reviews Neuroscience.

[3]  Thomas A. Carlson,et al.  Representational dynamics of object recognition: Feedforward and feedback information flows , 2016, NeuroImage.

[4]  Chris McNorgan,et al.  An attractor model of lexical conceptual processing: simulating semantic priming , 1999, Cogn. Sci..

[5]  David A. Tovar,et al.  Representational dynamics of object vision: the first 1000 ms. , 2013, Journal of vision.

[6]  Yaroslav O. Halchenko,et al.  The Animacy Continuum in the Human Ventral Vision Pathway , 2015, Journal of Cognitive Neuroscience.

[7]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[8]  D. F. Marks,et al.  An introduction , 1988, Experientia.

[9]  Alex Martin GRAPES—Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain , 2015, Psychonomic Bulletin & Review.

[10]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[11]  Elizabeth Jefferies,et al.  Semantic Processing in the Anterior Temporal Lobes: A Meta-analysis of the Functional Neuroimaging Literature , 2010, Journal of Cognitive Neuroscience.

[12]  Jerome Engel,et al.  The Temporal Lobe and Limbic System , 1998 .

[13]  Matthew A. Lambon Ralph,et al.  Convergent Connectivity and Graded Specialization in the Rostral Human Temporal Lobe as Revealed by Diffusion-Weighted Imaging Probabilistic Tractography , 2012, Journal of Cognitive Neuroscience.

[14]  F. Pulvermüller How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics , 2013, Trends in Cognitive Sciences.

[15]  T. Rogers,et al.  Anterior temporal cortex and semantic memory: Reconciling findings from neuropsychology and functional imaging , 2006, Cognitive, affective & behavioral neuroscience.

[16]  J. Hodges,et al.  Charting the progression in semantic dementia: implications for the organisation of semantic memory. , 1995 .

[17]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[18]  P. Dupont,et al.  Similarity of fMRI Activity Patterns in Left Perirhinal Cortex Reflects Semantic Similarity between Words , 2013, The Journal of Neuroscience.

[19]  Susan G. Wardle,et al.  Decoding the time-course of object recognition in the human brain: From visual features to categorical decisions , 2017, Neuropsychologia.

[20]  T. Rogers,et al.  The neural and computational bases of semantic cognition , 2016, Nature Reviews Neuroscience.

[21]  Antonio Torralba,et al.  Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence , 2016, Scientific Reports.

[22]  Angel Fernandez,et al.  A set of 254 Snodgrass-Vanderwart pictures standardized for Spanish: Norms for name agreement, image agreement, familiarity, and visual complexity , 1996 .

[23]  Christopher Barry,et al.  Naming the Snodgrass and Vanderwart Pictures: Effects of Age of Acquisition, Frequency, and Name Agreement , 1997 .

[24]  J. Hodges,et al.  Charting the progression in semantic dementia: implications for the organisation of semantic memory. , 1995, Memory.

[25]  E. Jefferies,et al.  Anterior temporal lobes mediate semantic representation: Mimicking semantic dementia by using rTMS in normal participants , 2007, Proceedings of the National Academy of Sciences.

[26]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[27]  J. G. Snodgrass,et al.  A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. , 1980 .

[28]  P. Gloor The Temporal Lobe and Limbic System , 1997 .

[29]  James L. McClelland,et al.  Parallel Distributed Processing at 25: Further Explorations in the Microstructure of Cognition , 2014, Cogn. Sci..

[30]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[31]  Alfonso Caramazza,et al.  NEUROPSYCHOLOGICAL AND NEUROIMAGING PERSPECTIVES ON CONCEPTUAL KNOWLEDGE: AN INTRODUCTION , 2003, Cognitive neuropsychology.

[32]  T. Rogers,et al.  Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. , 2006, Brain : a journal of neurology.

[33]  M. L. Lambon Ralph,et al.  Category-Specific versus Category-General Semantic Impairment Induced by Transcranial Magnetic Stimulation , 2010, Current Biology.

[34]  James L. McClelland,et al.  Semantic Cognition: A Parallel Distributed Processing Approach , 2004 .

[35]  Alex Martin,et al.  Semantic memory and the brain: structure and processes , 2001, Current Opinion in Neurobiology.

[36]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[37]  James L. McClelland,et al.  A computational model of semantic memory impairment: modality specificity and emergent category specificity. , 1991, Journal of experimental psychology. General.

[38]  T. Shallice,et al.  Perseverative and Semantic Influences on Visual Object Naming Errors in Optic Aphasia: A Connectionist Account , 1993, Journal of Cognitive Neuroscience.

[39]  D. Neary,et al.  Semantic dementia: a form of circumscribed cerebral atrophy , 1995 .

[40]  Jürgen Schmidhuber,et al.  Training Very Deep Networks , 2015, NIPS.

[41]  Mark S. Seidenberg,et al.  Computing the meanings of words in reading: cooperative division of labor between visual and phonological processes. , 2004, Psychological review.

[42]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[43]  Timothy T. Rogers,et al.  Connecting functional brain imaging and Parallel Distributed Processing , 2015 .

[44]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[45]  H. Fukuyama,et al.  The ‘when’ and ‘where’ of semantic coding in the anterior temporal lobe: Temporal representational similarity analysis of electrocorticogram data , 2016, Cortex.

[46]  Timothy T Rogers,et al.  Semantic memory is impaired in patients with unilateral anterior temporal lobe resection for temporal lobe epilepsy. , 2012, Brain : a journal of neurology.

[47]  Joel Z. Leibo,et al.  The dynamics of invariant object recognition in the human visual system. , 2014, Journal of neurophysiology.

[48]  S. Dehaene,et al.  Characterizing the dynamics of mental representations: the temporal generalization method , 2014, Trends in Cognitive Sciences.

[49]  Guy B. Williams,et al.  Atrophy, hypometabolism and white matter abnormalities in semantic dementia tell a coherent story. , 2011, Brain : a journal of neurology.

[50]  H Shibasaki,et al.  Subregions of human MT complex revealed by comparative MEG and direct electrocorticographic recordings , 2004, Clinical Neurophysiology.

[51]  M. Mesulam,et al.  From sensation to cognition. , 1998, Brain : a journal of neurology.

[52]  Nikolaus Kriegeskorte,et al.  Recurrence is required to capture the representational dynamics of the human visual system , 2019, Proceedings of the National Academy of Sciences.

[53]  James L. McClelland,et al.  A computational model of semantic memory impairment: modality specificity and emergent category specificity. , 1991 .

[54]  Radoslaw Martin Cichy,et al.  Resolving human object recognition in space and time , 2014, Nature Neuroscience.

[55]  A. Caramazza,et al.  Category-Specific Organization in the Human Brain Does Not Require Visual Experience , 2009, Neuron.

[56]  Alfonso Caramazza,et al.  Differential Activity for Animals and Manipulable Objects in the Anterior Temporal Lobes , 2011, Journal of Cognitive Neuroscience.

[57]  L. Tyler,et al.  Representational Similarity Analysis Reveals Commonalities and Differences in the Semantic Processing of Words and Objects , 2013, The Journal of Neuroscience.

[58]  Richard J. Binney,et al.  The ventral and inferolateral aspects of the anterior temporal lobe are crucial in semantic memory: evidence from a novel direct comparison of distortion-corrected fMRI, rTMS, and semantic dementia. , 2010, Cerebral cortex.

[59]  L. Tyler,et al.  Predicting the Time Course of Individual Objects with MEG , 2014, Cerebral cortex.

[60]  Dwight J. Kravitz,et al.  The ventral visual pathway: an expanded neural framework for the processing of object quality , 2013, Trends in Cognitive Sciences.

[61]  Hidenao Fukuyama,et al.  Left anterior temporal cortex actively engages in speech perception: A direct cortical stimulation study , 2011, Neuropsychologia.

[62]  T. Rogers,et al.  A unified model of human semantic knowledge and its disorders , 2017, Nature Human Behaviour.

[63]  T. Allison,et al.  Word recognition in the human inferior temporal lobe , 1994, Nature.