Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream

A significant body of research in cognitive neuroscience is aimed at understanding how object concepts are represented in the human brain. However, it remains unknown whether and where the visual and abstract conceptual features that define an object concept are integrated. We addressed this issue by comparing the neural pattern similarities among object-evoked fMRI responses with behavior-based models that independently captured the visual and conceptual similarities among these stimuli. Our results revealed evidence for distinctive coding of visual features in lateral occipital cortex, and conceptual features in the temporal pole and parahippocampal cortex. By contrast, we found evidence for integrative coding of visual and conceptual object features in perirhinal cortex. The neuroanatomical specificity of this effect was highlighted by results from a searchlight analysis. Taken together, our findings suggest that perirhinal cortex uniquely supports the representation of fully specified object concepts through the integration of their visual and conceptual features.

[1]  C. Ranganath,et al.  Functional connectivity based parcellation of the human medial temporal lobe , 2016, Neurobiology of Learning and Memory.

[2]  Stephen M Smith,et al.  Fast robust automated brain extraction , 2002, Human brain mapping.

[3]  Edward B. O'Neil,et al.  Perirhinal Cortex Contributes to Accuracy in Recognition Memory and Perceptual Discriminations , 2009, The Journal of Neuroscience.

[4]  Fei-Fei Li,et al.  Differential connectivity within the Parahippocampal Place Area , 2013, NeuroImage.

[5]  Demis Hassabis,et al.  Semantic representations in the temporal pole predict false memories , 2016, Proceedings of the National Academy of Sciences.

[6]  J. Hodges,et al.  Semantic dementia. Progressive fluent aphasia with temporal lobe atrophy. , 1992, Brain : a journal of neurology.

[7]  James L. McClelland,et al.  Semantic Cognition: A Parallel Distributed Processing Approach , 2004 .

[8]  Bruce Fischl,et al.  Accurate and robust brain image alignment using boundary-based registration , 2009, NeuroImage.

[9]  Elizabeth Jefferies,et al.  Fractionating the anterior temporal lobe: MVPA reveals differential responses to input and conceptual modality , 2017, NeuroImage.

[10]  Alex Martin,et al.  Semantic memory and the brain: structure and processes , 2001, Current Opinion in Neurobiology.

[11]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[12]  T. Rogers,et al.  The neural and computational bases of semantic cognition , 2016, Nature Reviews Neuroscience.

[13]  Antonio R. Damasio,et al.  The Brain Binds Entities and Events by Multiregional Activation from Convergence Zones , 1989, Neural Computation.

[14]  Andy C. H. Lee,et al.  Going beyond LTM in the MTL: A synthesis of neuropsychological and neuroimaging findings on the role of the medial temporal lobe in memory and perception , 2010, Neuropsychologia.

[15]  M. Bar.  Visual objects in context , 2004, Nature Reviews Neuroscience.

[16]  Morgan D. Barense,et al.  Conjunctive Coding of Complex Object Features. , 2016, Cerebral cortex.

[17]  M. Kiefer,et al.  Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions , 2012, Cortex.

[18]  E. Luders,et al.  Voxel-Based Morphometry , 2015 .

[19]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[20]  T. Shallice,et al.  Category specific semantic impairments. , 1998, Brain : a journal of neurology.

[21]  M. Goodale,et al.  The visual brain in action , 1995 .

[22]  Justin L. Vincent,et al.  Distinct cortical anatomy linked to subregions of the medial temporal lobe revealed by intrinsic functional connectivity. , 2008, Journal of neurophysiology.

[23]  T. Rogers,et al.  A unified model of human semantic knowledge and its disorders , 2017, Nature Human Behaviour.

[24]  Alex Martin.  GRAPES—Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain , 2015, Psychonomic Bulletin & Review.

[25]  M. Bar,et al.  Cortical Analysis of Visual Context , 2003, Neuron.

[26]  C. Ranganath,et al.  Two cortical systems for memory-guided behaviour , 2012, Nature Reviews Neuroscience.

[27]  E. Jefferies,et al.  Anterior temporal lobes mediate semantic representation: Mimicking semantic dementia by using rTMS in normal participants , 2007, Proceedings of the National Academy of Sciences.

[28]  Nikolaus Kriegeskorte,et al.  Comparison of multivariate classifiers and response normalizations for pattern-information fMRI , 2010, NeuroImage.

[29]  Stephen M. Smith,et al.  Threshold-free cluster enhancement: Addressing problems of smoothing, threshold dependence and localisation in cluster inference , 2009, NeuroImage.

[30]  Russell A. Epstein,et al.  Outside Looking In: Landmark Generalization in the Human Navigational System , 2015, The Journal of Neuroscience.

[31]  Andy C. H. Lee,et al.  Behavioral / Systems / Cognitive Functional Specialization in the Human Medial Temporal Lobe , 2005 .

[32]  Edward B. O'Neil,et al.  Distinct Familiarity-Based Response Patterns for Faces and Buildings in Perirhinal and Parahippocampal Cortex , 2013, The Journal of Neuroscience.

[33]  Yuji Naya,et al.  The perirhinal cortex. , 2014, Annual review of neuroscience.

[34]  A. Caramazza,et al.  Brain Regions That Represent Amodal Conceptual Knowledge , 2013, The Journal of Neuroscience.

[35]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Kara L. Agster,et al.  Cortical efferents of the perirhinal, postrhinal, and entorhinal cortices of the rat , 2009, Hippocampus.

[37]  Richard N. A. Henson,et al.  Perception and Conception: Temporal Lobe Activity during Complex Discriminations of Familiar and Novel Faces and Objects , 2011, Journal of Cognitive Neuroscience.

[38]  Mark S. Seidenberg,et al.  Semantic feature production norms for a large set of living and nonliving things , 2005, Behavior research methods.

[39]  N. Kriegeskorte,et al.  Author ' s personal copy Representational geometry : integrating cognition , computation , and the brain , 2013 .

[40]  Morgan D. Barense,et al.  The human medial temporal lobe processes online representations of complex objects , 2007, Neuropsychologia.

[41]  Ricardo Insausti,et al.  Identification of the human medial temporal lobe regions on magnetic resonance images , 2014, Human brain mapping.

[42]  Jessey Wright,et al.  How landmark suitability shapes recognition memory signals for objects in the medial temporal lobes , 2018, NeuroImage.

[43]  S. Edelman,et al.  Differential Processing of Objects under Various Viewing Conditions in the Human Lateral Occipital Complex , 1999, Neuron.

[44]  L. Tyler,et al.  Representational Similarity Analysis Reveals Commonalities and Differences in the Semantic Processing of Words and Objects , 2013, The Journal of Neuroscience.

[45]  Rosemary A. Cowell,et al.  Distributed category‐specific recognition‐memory signals in human perirhinal cortex , 2016, Hippocampus.

[46]  B. Sahakian,et al.  Differing patterns of temporal atrophy in Alzheimer’s disease and semantic dementia , 2001, Neurology.

[47]  Differentiating Roles , 1978 .

[48]  M. Bar,et al.  The parahippocampal cortex mediates spatial and nonspatial associations. , 2007, Cerebral cortex.

[49]  T. Rogers,et al.  Anterior temporal cortex and semantic memory: Reconciling findings from neuropsychology and functional imaging , 2006, Cognitive, affective & behavioral neuroscience.

[50]  Keiji Tanaka,et al.  Neural representation for object recognition in inferotemporal cortex , 2016, Current Opinion in Neurobiology.

[51]  Daniel Tranel,et al.  The left temporal pole is important for retrieving words for unique concrete entities , 2009, Aphasiology.

[52]  D. Amaral,et al.  Perirhinal and parahippocampal cortices of the macaque monkey: Cortical afferents , 1994, The Journal of comparative neurology.

[53]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.

[54]  Morgan D. Barense,et al.  Influence of Conceptual Knowledge on Visual Object Discrimination : Insights from Semantic Dementia and MTL Amnesia , 2010 .

[55]  L. Tyler,et al.  Object-Specific Semantic Coding in Human Perirhinal Cortex , 2014, The Journal of Neuroscience.

[56]  Uta Noppeney,et al.  Temporal lobe lesions and semantic impairment: a comparison of herpes simplex virus encephalitis and semantic dementia. , 2006, Brain : a journal of neurology.

[57]  David Gaffan,et al.  Perirhinal cortical contributions to object perception , 2006, Trends in Cognitive Sciences.

[58]  James V. Haxby,et al.  CoSMoMVPA: Multi-Modal Multivariate Pattern Analysis of Neuroimaging Data in Matlab/GNU Octave , 2016, bioRxiv.

[59]  M. Bar,et al.  The role of the parahippocampal cortex in cognition , 2013, Trends in Cognitive Sciences.

[60]  Leslie G. Ungerleider,et al.  Discrete Cortical Regions Associated with Knowledge of Color and Knowledge of Action , 1995, Science.

[61]  E. Warrington.  Quarterly Journal of Experimental Psychology the Selective Impairment of Semantic Memory the Selective Impairment of Semantic Memory , 2022 .

[62]  Morgan D Barense,et al.  The perirhinal cortex modulates V2 activity in response to the agreement between part familiarity and configuration familiarity , 2012, Hippocampus.

[63]  T. Bussey,et al.  Perceptual–mnemonic functions of the perirhinal cortex , 1999, Trends in Cognitive Sciences.

[64]  L. Saksida,et al.  Visual perception and memory: a new view of medial temporal lobe function in primates and rodents. , 2007, Annual review of neuroscience.

[65]  Li Su,et al.  A Toolbox for Representational Similarity Analysis , 2014, PLoS Comput. Biol..

[66]  Lorraine K. Tyler,et al.  When leopards lose their spots: knowledge of visual properties in category-specific deficits for living things , 1997 .

[67]  David C. Van Essen,et al.  Application of Information Technology: An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex , 2001, J. Am. Medical Informatics Assoc..

[68]  Richard S. J. Frackowiak,et al.  A voxel‐based morphometry study of semantic dementia: Relationship between temporal lobe atrophy and semantic memory , 2000, Annals of neurology.

[69]  Rutvik H. Desai,et al.  The neurobiology of semantic memory , 2011, Trends in Cognitive Sciences.

[70]  Marieke Mur.  What's the Difference between a Tiger and a Cat? From Visual Object to Semantic Concept via the Perirhinal Cortex , 2014, The Journal of Neuroscience.

[71]  M. L. Lambon Ralph,et al.  Conceptual knowledge is underpinned by the temporal pole bilaterally: convergent evidence from rTMS. , 2009, Cerebral cortex.

[72]  P. Dupont,et al.  Similarity of fMRI Activity Patterns in Left Perirhinal Cortex Reflects Semantic Similarity between Words , 2013, The Journal of Neuroscience.

[73]  Andy C. H. Lee,et al.  Intact Memory for Irrelevant Information Impairs Perception in Amnesia , 2012, Neuron.

[74]  Nancy Kanwisher,et al.  A cortical representation of the local visual environment , 1998, Nature.

[75]  T. Rogers,et al.  A duck with four legs: Investigating the structure of conceptual knowledge using picture drawing in semantic dementia , 2003, Cognitive neuropsychology.

[76]  J. Hodges,et al.  The relationship between comprehension and oral reading in progressive fluent aphasia , 1994, Neuropsychologia.

[77]  Alexis Amadon,et al.  Word meaning in the ventral visual path: a perceptual to conceptual gradient of semantic coding , 2016, NeuroImage.

[78]  Yi Chen,et al.  Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size control , 2011, NeuroImage.

[79]  D. A. Walker,et al.  JMASM9: Converting Kendall’s Tau For Correlational Or Meta-Analytic Analyses , 2003 .

[80]  Z Kourtzi,et al.  Representation of Perceived Object Shape by the Human Lateral Occipital Complex , 2001, Science.

[81]  Alan C. Evans,et al.  Volumetry of temporopolar, perirhinal, entorhinal and parahippocampal cortex from high-resolution MR images: considering the variability of the collateral sulcus. , 2002, Cerebral cortex.

[82]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[83]  Susanne Ferber,et al.  Stimulus familiarity modulates functional connectivity of the perirhinal cortex and anterior hippocampus during visual discrimination of faces and objects , 2014, Front. Hum. Neurosci..

[84]  Andy C. H. Lee,et al.  Specialization in the medial temporal lobe for processing of objects and scenes , 2005, Hippocampus.

[85]  D. V. van Essen,et al.  A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex. , 2005, NeuroImage.

[86]  Billi Randall,et al.  The perirhinal cortex and conceptual processing: Effects of feature-based statistics following damage to the anterior temporal lobes , 2015, Neuropsychologia.

[87]  J. Rodd,et al.  Anteromedial temporal cortex supports fine-grained differentiation among objects. , 2005, Cerebral cortex.

[88]  L. Tyler,et al.  Contrasting effects of feature-based statistics on the categorisation and basic-level identification of visual objects , 2012, Cognition.

[89]  Ken McRae,et al.  Perirhinal cortex tracks degree of recent as well as cumulative lifetime experience with object concepts , 2017, Cortex.

[90]  C. Price,et al.  Perirhinal Contributions to Human Visual Perception , 2007, Current Biology.

[91]  Michael Brady,et al.  Improved Optimization for the Robust and Accurate Linear Registration and Motion Correction of Brain Images , 2002, NeuroImage.

[92]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[93]  Mark W. Woolrich,et al.  Advances in functional and structural MR image analysis and implementation as FSL , 2004, NeuroImage.

[94]  Tianzi Jiang,et al.  Connectivity Profiles Reveal a Transition Subarea in the Parahippocampal Region That Integrates the Anterior Temporal–Posterior Medial Systems , 2016, The Journal of Neuroscience.

[95]  Andy C. H. Lee,et al.  Differentiating the Roles of the Hippocampus and Perirhinal Cortex in Processes beyond Long-Term Declarative Memory: A Double Dissociation in Dementia , 2006, The Journal of Neuroscience.

[96]  R. Malach,et al.  Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[97]  S. Thompson-Schill.  Neuroimaging studies of semantic memory: inferring “how” from “where” , 2003, Neuropsychologia.

[98]  Richard J. Binney,et al.  The ventral and inferolateral aspects of the anterior temporal lobe are crucial in semantic memory: evidence from a novel direct comparison of distortion-corrected fMRI, rTMS, and semantic dementia. , 2010, Cerebral cortex.

[99]  A. Caramazza,et al.  Conceptual Object Representations in Human Anterior Temporal Cortex , 2012, The Journal of Neuroscience.

[100]  L. Tyler,et al.  Towards a distributed account of conceptual knowledge , 2001, Trends in Cognitive Sciences.

[101]  L. Tyler,et al.  Understanding What We See: How We Derive Meaning From Vision , 2015, Trends in Cognitive Sciences.

[102]  David C. Van Essen,et al.  A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex , 2005, NeuroImage.