How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge

How conceptual knowledge is represented in the human brain remains to be determined. To address the differential role of low-level sensory-based and high-level abstract features in semantic processing, we combined behavioral studies of linguistic production and brain activity measures by functional magnetic resonance imaging in sighted and congenitally blind individuals while they performed a property-generation task with concrete nouns from eight categories, presented through visual and/or auditory modalities. Patterns of neural activity within a large semantic cortical network that comprised parahippocampal, lateral occipital, temporo-parieto-occipital and inferior parietal cortices correlated with linguistic production and were independent both from the modality of stimulus presentation (either visual or auditory) and the (lack of) visual experience. In contrast, selected modality-dependent differences were observed only when the analysis was limited to the individual regions within the semantic cortical network. We conclude that conceptual knowledge in the human brain relies on a distributed, modality-independent cortical representation that integrates the partial category and modality specific information retained at a regional level.

[1]  Emiliano Ricciardi,et al.  Beyond sensory images: Object-based representation in the human ventral pathway. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[2]  Michael Gribskov,et al.  Combining evidence using p-values: application to sequence homology searches , 1998, Bioinform..

[3]  Philip A. Cook,et al.  Category-specific semantic memory: Converging evidence from bold fMRI and Alzheimer's disease , 2013, NeuroImage.

[4]  R. Saxe,et al.  Language processing in the occipital cortex of congenitally blind adults , 2011, Proceedings of the National Academy of Sciences.

[5]  Thomas Serre,et al.  Reading the mind's eye: Decoding category information during mental imagery , 2010, NeuroImage.

[6]  Joseph T. Devlin,et al.  Supramarginal gyrus involvement in visual word recognition , 2009, Cortex.

[7]  Jung-Kyong Kim,et al.  Tactile–Auditory Shape Learning Engages the Lateral Occipital Complex , 2011, The Journal of Neuroscience.

[8]  Riitta Salmelin,et al.  Tracking neural coding of perceptual and semantic features of concrete nouns , 2012, NeuroImage.

[9]  Nikolaus Kriegeskorte,et al.  Reaction Time for Object Categorization Is Predicted by Representational Distance , 2014, Journal of Cognitive Neuroscience.

[10]  Alessandro Lenci,et al.  BLIND: a set of semantic feature norms from the congenitally blind , 2013, Behavior research methods.

[11]  L. Tyler,et al.  Representational Similarity Analysis Reveals Commonalities and Differences in the Semantic Processing of Words and Objects , 2013, The Journal of Neuroscience.

[12]  Bradford Z. Mahon,et al.  The organization of conceptual knowledge: the evidence from category-specific semantic deficits , 2003, Trends in Cognitive Sciences.

[13]  A. Caramazza,et al.  How Visual Is the Visual Cortex? Comparing Connectional and Functional Fingerprints between Congenitally Blind and Sighted Individuals , 2015, The Journal of Neuroscience.

[14]  Friedemann Pulvermüller,et al.  Understanding in an instant: Neurophysiological evidence for mechanistic language circuits in the brain , 2009, Brain and Language.

[15]  Alfonso Caramazza,et al.  The Representation of Tools in Left Parietal Cortex Is Independent of Visual Experience , 2010, Psychological science.

[16]  E. Maguire,et al.  What does the retrosplenial cortex do? , 2009, Nature Reviews Neuroscience.

[17]  Tom M. Mitchell,et al.  Learning to Decode Cognitive States from Brain Images , 2004, Machine Learning.

[18]  N. Kriegeskorte,et al.  Author ' s personal copy Representational geometry : integrating cognition , computation , and the brain , 2013 .

[19]  Alfonso Caramazza,et al.  Category-selective neural substrates for person- and place-related concepts , 2013, Cortex.

[20]  L. Gleitman,et al.  Language and Experience: Evidence from the Blind Child , 1988 .

[21]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[22]  Andrew C. Connolly,et al.  Effect of congenital blindness on the semantic representation of some everyday concepts , 2007, Proceedings of the National Academy of Sciences.

[23]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[24]  Tom M. Mitchell,et al.  Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation , 2011, NeuroImage.

[25]  G. Rizzolatti,et al.  Parietal Lobe: From Action Organization to Intention Understanding , 2005, Science.

[26]  Cathy J. Price,et al.  A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading , 2012, NeuroImage.

[27]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[28]  Mark S. Seidenberg,et al.  Concept Representation Reflects Multimodal Abstraction: A Framework for Embodied Semantics. , 2016, Cerebral cortex.

[29]  P. Dupont,et al.  Similarity of fMRI Activity Patterns in Left Perirhinal Cortex Reflects Semantic Similarity between Words , 2013, The Journal of Neuroscience.

[30]  Alexander G. Huth,et al.  Attention During Natural Vision Warps Semantic Representation Across the Human Brain , 2013, Nature Neuroscience.

[31]  S. Thompson-Schill,et al.  Reworking the language network , 2014, Trends in Cognitive Sciences.

[32]  T. Shallice,et al.  Category specific semantic impairments. , 1998, Brain : a journal of neurology.

[33]  Francisco Pereira,et al.  Using Wikipedia to learn semantic feature representations of concrete concepts in neuroimaging experiments , 2013, Artif. Intell..

[34]  Russell A. Poldrack,et al.  The impact of study design on pattern estimation for single-trial multivariate pattern analysis , 2014, NeuroImage.

[35]  Soojin Park,et al.  Disentangling Scene Content from Spatial Boundary: Complementary Roles for the Parahippocampal Place Area and Lateral Occipital Complex in Representing Real-World Scenes , 2011, The Journal of Neuroscience.

[36]  M. Bedny,et al.  “Visual” Cortex of Congenitally Blind Adults Responds to Syntactic Movement , 2015, The Journal of Neuroscience.

[37]  Guido Gainotti,et al.  The influence of anatomical locus of lesion and of gender-related familiarity factors in category-specific semantic disorders for animals, fruits and vegetables: A review of single-case studies , 2010, Cortex.

[38]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[39]  Simon B. Eickhoff,et al.  Assignment of functional activations to probabilistic cytoarchitectonic areas revisited , 2007, NeuroImage.

[40]  J. Diedrichsen,et al.  Hand use predicts the structure of representations in sensorimotor cortex , 2015, Nature Neuroscience.

[41]  P. Pietrini,et al.  A topographical organization for action representation in the human brain , 2015, Human brain mapping.

[42]  Joseph T. Devlin,et al.  The emergence of category specific deficits in a distributed semantic system , 2002 .

[43]  Xiaoping P. Hu,et al.  fMRI evidence for word association and situated simulation in conceptual processing , 2008, Journal of Physiology-Paris.

[44]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[45]  J. Rauschecker,et al.  Preserved Functional Specialization for Spatial Processing in the Middle Occipital Gyrus of the Early Blind , 2010, Neuron.

[46]  A. Caramazza,et al.  Multivoxel Pattern Analysis Reveals Auditory Motion Information in MT+ of Both Congenitally Blind and Sighted Individuals , 2013, PloS one.

[47]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[48]  A. Caramazza,et al.  Functional connectivity of visual cortex in the blind follows retinotopic organization principles , 2015, Brain : a journal of neurology.

[49]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[50]  Alfonso Caramazza,et al.  Tool Selectivity in Left Occipitotemporal Cortex Develops without Vision , 2013, Journal of Cognitive Neuroscience.

[51]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[52]  Matthew H. Davis,et al.  Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI , 2002, Neuropsychologia.

[53]  T. Rogers,et al.  Object categorization: reversals and explanations of the basic-level advantage. , 2007, Journal of experimental psychology. General.

[54]  L. Squire,et al.  The anatomy of semantic knowledge: medial vs. lateral temporal lobe. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[55]  A. Caramazza,et al.  Nonvisual and Visual Object Shape Representations in Occipitotemporal Cortex: Evidence from Congenitally Blind and Sighted Adults , 2014, The Journal of Neuroscience.

[56]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[57]  P. Hoffman,et al.  The Roles of Left Versus Right Anterior Temporal Lobes in Conceptual Knowledge: An ALE Meta-analysis of 97 Functional Neuroimaging Studies , 2015, Cerebral cortex.

[58]  William M. Stern,et al.  Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex , 2007, Nature Neuroscience.

[59]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[60]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[61]  J. S. Guntupalli,et al.  The Representation of Biological Classes in the Human Brain , 2012, The Journal of Neuroscience.

[62]  Robert Oostenveld,et al.  Modality-independent decoding of semantic information from the human brain. , 2014, Cerebral cortex.

[63]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[64]  J. S. Guntupalli,et al.  Decoding neural representational spaces using multivariate pattern analysis. , 2014, Annual review of neuroscience.

[65]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[66]  E. Warrington,et al.  Categories of knowledge. Further fractionations and an attempted integration. , 1987, Brain : a journal of neurology.

[67]  Karl J. Friston,et al.  Effects of visual deprivation on the organization of the semantic system. , 2003, Brain : a journal of neurology.

[68]  Mark W. Woolrich,et al.  Advances in functional and structural MR image analysis and implementation as FSL , 2004, NeuroImage.

[69]  A. Caramazza,et al.  Category-Specific Organization in the Human Brain Does Not Require Visual Experience , 2009, Neuron.

[70]  Emily J. Ward,et al.  How reliable are visual context effects in the parahippocampal place area? , 2010, Cerebral cortex.