Multisensory Part-based Representations of Objects in Human Lateral Occipital Cortex

The format of high-level object representations in temporal-occipital cortex is a fundamental and as yet unresolved issue. Here we use fMRI to show that human lateral occipital cortex (LOC) encodes novel 3-D objects in a multisensory and part-based format. We show that visual and haptic exploration of objects leads to similar patterns of neural activity in human LOC and that the shared variance between visually and haptically induced patterns of BOLD contrast in LOC reflects the part structure of the objects. We also show that linear classifiers trained on neural data from LOC on a subset of the objects successfully predict a novel object based on its component part structure. These data demonstrate a multisensory code for object representations in LOC that specifies the part structure of objects.

[1]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[2]  G. Humphreys,et al.  A case of integrative visual agnosia. , 1987, Brain : a journal of neurology.

[3]  J. Talairach,et al.  Co-Planar Stereotaxic Atlas of the Human Brain: 3-Dimensional Proportional System: An Approach to Cerebral Imaging , 1988 .

[4]  H H Bülthoff,et al.  Psychophysical support for a two-dimensional view interpolation theory of object recognition. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[5]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[6]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[7]  M. Tarr.  Visual Object Recognition: Can A Single Mechanism Suffice? , 1998 .

[8]  T. Hendler,et al.  Visuo-haptic object-related activation in the ventral visual pathway , 2001, Nature Neuroscience.

[9]  Z Kourtzi,et al.  Representation of Perceived Object Shape by the Human Lateral Occipital Complex , 2001, Science.

[10]  N. Kanwisher,et al.  The lateral occipital complex and its role in object recognition , 2001, Vision Research.

[11]  T. Hendler,et al.  Convergence of visual and tactile shape processing in the human lateral occipital complex. , 2002, Cerebral cortex.

[12]  Ravi S. Menon,et al.  Haptic study of three-dimensional objects activates extrastriate visual areas , 2002, Neuropsychologia.

[13]  Hideko F. Norman,et al.  The visual and haptic perception of natural object shape , 2004, Perception & Psychophysics.

[14]  I. Biederman,et al.  Neural evidence for intermediate representations in object recognition , 2006, Vision Research.

[15]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Heinrich H. Bülthoff,et al.  Object Feature Validation Using Visual and Haptic Similarity Ratings , 2022 .

[17]  H. Bülthoff,et al.  Multimodal similarity and categorization of novel, three-dimensional objects , 2007, Neuropsychologia.

[18]  M. Tarr,et al.  Visual object recognition: do we know more now than we did 20 years ago? , 2007, Annual review of psychology.

[19]  S. Lacey,et al.  Cross-Modal Object Recognition Is Viewpoint-Independent , 2007, PloS one.

[20]  R. Lawson.  A comparison of the effects of depth rotation on visual and haptic three-dimensional object recognition. , 2009, Journal of experimental psychology. Human perception and performance.

[21]  H. Bülthoff,et al.  Visual and haptic perceptual spaces show high similarity in humans. , 2010, Journal of vision.

[22]  J. Kaiser,et al.  Visuohaptic convergence in a corticocerebellar network , 2010, The European journal of neuroscience.

[23]  H. Bülthoff,et al.  Similarity and categorization: from vision to touch. , 2011, Acta psychologica.

[24]  I. Biederman,et al.  Neural encoding of relative position. , 2011, Journal of experimental psychology. Human perception and performance.

[25]  Christian Wallraven,et al.  Categorizing natural objects: a comparison of the visual and the haptic modalities , 2011, Experimental Brain Research.

[26]  Jens Schwarzbach,et al.  A simple framework (ASF) for behavioral and neuroimaging experiments based on the psychophysics toolbox for MATLAB , 2011, Behavior research methods.

[27]  A. Caramazza,et al.  Nonvisual and Visual Object Shape Representations in Occipitotemporal Cortex: Evidence from Congenitally Blind and Sighted Adults , 2014, The Journal of Neuroscience.

[28]  Bradford Z. Mahon,et al.  A bimodal tuning curve for spatial frequency across left and right human orbital frontal cortex during object recognition. , 2014, Cerebral cortex.

[29]  J. Kaiser,et al.  Enhanced visuo-haptic integration for the non-dominant hand , 2015, Brain Research.

[30]  Radoslaw Martin Cichy,et al.  Non-holistic coding of objects in lateral occipital complex with and without attention , 2015, NeuroImage.

[31]  Robert A. Jacobs,et al.  From Sensory Signals to Modality-Independent Conceptual Representations: A Probabilistic Language of Thought Approach , 2015, PLoS Comput. Biol..

[32]  Bradford Z. Mahon,et al.  The Representation of Object-Directed Action and Function Knowledge in the Human Brain. , 2016, Cerebral cortex.