Multimodal representations of person identity individuated with fMRI

Recognizing the identity of a person is fundamental to guide social interactions. We can recognize the identity of a person looking at her face, but also listening to her voice. An important question concerns how visual and auditory information come together, enabling us to recognize identity independently of the modality of the stimulus. This study reports converging evidence across univariate contrasts and multivariate classification showing that the posterior superior temporal sulcus (pSTS), previously known to encode polymodal visual and auditory representations, encodes information about person identity with invariance within and across modality. In particular, pSTS shows selectivity for faces, selectivity for voices, classification of face identity across image transformations within the visual modality, and classification of person identity across modality.

[1]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[2]  D. Pandya,et al.  Parietal, temporal, and occipita projections to cortex of the superior temporal sulcus in the rhesus monkey: A retrograde tracer study , 1994, The Journal of comparative neurology.

[3]  Galia Avidan,et al.  Reduced structural connectivity in ventral visual cortex in congenital prosopagnosia , 2009, Nature Neuroscience.

[4]  R. Zatorre,et al.  Voice-selective areas in human auditory cortex , 2000, Nature.

[5]  Alfonso Caramazza,et al.  The neural mechanisms for the recognition of face identity in humans , 2014, Front. Psychol..

[6]  Béatrice de Gelder,et al.  Lateralization for dynamic facial expressions in human superior temporal sulcus , 2015, NeuroImage.

[7]  A. Caramazza,et al.  Decoding representations of face identity that are tolerant to rotation. , 2014, Cerebral cortex.

[8]  Karalyn Patterson,et al.  Left/right asymmetry of atrophy in semantic dementia , 2003, Neurology.

[9]  M. Chun,et al.  Selecting and perceiving multiple visual objects , 2009, Trends in Cognitive Sciences.

[10]  M Zaitsev,et al.  Point spread function mapping with parallel imaging techniques and high acceleration factors: Fast, robust, and flexible method for echo‐planar imaging distortion correction , 2004, Magnetic resonance in medicine.

[11]  Nicholas B. Turk-Browne,et al.  Representations of individuals in ventral temporal cortex defined by faces and biographies , 2013, Neuropsychologia.

[12]  Á. Pascual-Leone,et al.  Repetitive TMS over posterior STS disrupts perception of biological motion , 2005, Vision Research.

[13]  M. Peelen,et al.  Supramodal Representations of Perceived Emotions in the Human Brain , 2010, The Journal of Neuroscience.

[14]  J. Hodges,et al.  Non-verbal semantic impairment in semantic dementia , 2000, Neuropsychologia.

[15]  Alfonso Caramazza,et al.  From Parts to Identity: Invariance and Sensitivity of Face Representations to Different Face Halves. , 2016, Cerebral cortex.

[16]  Guido Gainotti,et al.  Slowly progressive defect in recognition of familiar people in a patient with right anterior temporal atrophy. , 2003, Brain : a journal of neurology.

[17]  Pascal Belin,et al.  Crossmodal Adaptation in Right Posterior Superior Temporal Sulcus during Face–Voice Emotional Integration , 2014, The Journal of Neuroscience.

[18]  R. Blake,et al.  Brain Areas Involved in Perception of Biological Motion , 2000, Journal of Cognitive Neuroscience.

[19]  H. Kashima,et al.  A deficit in discriminating gaze direction in a case with right superior temporal gyrus lesion , 2006, Neuropsychologia.

[20]  Andreas Kleinschmidt,et al.  Interaction of Face and Voice Areas during Speaker Recognition , 2005, Journal of Cognitive Neuroscience.

[21]  J. Hodges,et al.  Semantic dementia. Progressive fluent aphasia with temporal lobe atrophy. , 1992 .

[22]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[23]  Yaoda Xu,et al.  Representing connected and disconnected shapes in human inferior intraparietal sulcus , 2008, NeuroImage.

[24]  Alice J. O'Toole,et al.  Dissociable Neural Patterns of Facial Identity across Changes in Viewpoint , 2010, Journal of Cognitive Neuroscience.

[25]  A. Damasio,et al.  A neural basis for the retrieval of conceptual knowledge , 1997, Neuropsychologia.

[26]  Ingrid R. Olson,et al.  Beyond the FFA: The role of the ventral anterior temporal lobes in face processing , 2014, Neuropsychologia.

[27]  Elizabeth Jefferies,et al.  Semantic Processing in the Anterior Temporal Lobes: A Meta-analysis of the Functional Neuroimaging Literature , 2010, Journal of Cognitive Neuroscience.

[28]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[29]  Yaoda Xu,et al.  Visual grouping in human parietal cortex , 2007, Proceedings of the National Academy of Sciences.

[30]  R. Goebel,et al.  Individual faces elicit distinct response patterns in human anterior temporal cortex , 2007, Proceedings of the National Academy of Sciences.

[31]  M. Chun,et al.  Dissociable neural mechanisms supporting visual short-term memory for objects , 2006, Nature.

[32]  Keiji Tanaka,et al.  Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. , 1988, Journal of neurophysiology.

[33]  R. Zatorre,et al.  Human temporal-lobe response to vocal sounds. , 2002, Brain research. Cognitive brain research.

[34]  Rainer Goebel,et al.  Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns , 2008, NeuroImage.

[35]  J. Sergent,et al.  Functional neuroanatomy of face and object processing. A positron emission tomography study. , 1992, Brain : a journal of neurology.

[36]  A. Kleinschmidt,et al.  Modulation of neural responses to speech by directing attention to voices or verbal content. , 2003, Brain research. Cognitive brain research.

[37]  Yaoda Xu The Role of the Superior Intraparietal Sulcus in Supporting Visual Short-Term Memory for Multifeature Objects , 2007, The Journal of Neuroscience.

[38]  L. Benevento,et al.  Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey , 1977, Experimental Neurology.

[39]  Marlene Behrmann,et al.  Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis , 2011, Proceedings of the National Academy of Sciences.

[40]  S. Edelman,et al.  Differential Processing of Objects under Various Viewing Conditions in the Human Lateral Occipital Complex , 1999, Neuron.

[41]  M. Tarr,et al.  The Fusiform Face Area is Part of a Network that Processes Faces at the Individual Level , 2000, Journal of Cognitive Neuroscience.

[42]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[43]  Rebecca Saxe,et al.  A Common Neural Code for Perceived and Inferred Emotion , 2014, The Journal of Neuroscience.

[44]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[45]  Brice A. Kuhl,et al.  Neural portraits of perception: Reconstructing face images from evoked brain activity , 2014, NeuroImage.

[46]  H. P. Op de Beeck,et al.  Representations of Facial Identity Information in the Ventral Visual Stream Investigated with Multivoxel Pattern Analyses , 2013, The Journal of Neuroscience.

[47]  T. Singer,et al.  The role of anterior insular cortex in social emotions , 2010, Brain Structure and Function.

[48]  N. Sadato,et al.  Attention to emotion modulates fMRI activity in human right superior temporal sulcus. , 2001, Brain research. Cognitive brain research.

[49]  R. Dolan,et al.  fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. , 2004, Journal of neurophysiology.

[50]  Alfred Anwander,et al.  Direct Structural Connections between Voice- and Face-Recognition Areas , 2011, The Journal of Neuroscience.

[51]  D. Neary,et al.  Semantic dementia: a form of circumscribed cerebral atrophy , 1995 .

[52]  Richard D. Hichwa,et al.  A neural basis for lexical retrieval , 1996, Nature.

[53]  J. Barton,et al.  Voice Recognition in Face-Blind Patients. , 2016, Cerebral cortex.

[54]  D. Neary,et al.  Knowledge of famous faces and names in semantic dementia. , 2004, Brain : a journal of neurology.

[55]  Maria Luisa Gorno-Tempini,et al.  Cognitive and Behavioral Profile in a Case of Right Anterior Temporal Lobe Neurodegeneration , 2004, Cortex.

[56]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[57]  J. C. Meadows The anatomical basis of prosopagnosia , 1974, Journal of neurology, neurosurgery, and psychiatry.

[58]  Simona Luzzi,et al.  Semantic memory is an amodal, dynamic system: Evidence from the interaction of naming and object use in semantic dementia , 2004, Cognitive neuropsychology.

[59]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[60]  Nikolaus Kriegeskorte,et al.  Face-identity change activation outside the face system: "release from adaptation" may not always indicate neuronal selectivity. , 2010, Cerebral cortex.

[61]  Richard S. J. Frackowiak,et al.  A voxel‐based morphometry study of semantic dementia: Relationship between temporal lobe atrophy and semantic memory , 2000, Annals of neurology.

[62]  K. Luan Phan,et al.  Functional Neuroimaging Studies of Human Emotions , 2004, CNS Spectrums.

[63]  Doris Y. Tsao,et al.  Single-Unit Recordings in the Macaque Face Patch System Reveal Limitations of fMRI MVPA , 2015, The Journal of Neuroscience.

[64]  A. Young,et al.  Understanding the recognition of facial identity and facial expression , 2005, Nature Reviews Neuroscience.

[65]  Daniel D. Dilks,et al.  Differential selectivity for dynamic versus static information in face-selective cortical regions , 2011, NeuroImage.

[66]  R. Tootell,et al.  An anterior temporal face patch in human cortex, predicted by macaque maps , 2009, Proceedings of the National Academy of Sciences.

[67]  P. Sinha,et al.  Functional neuroanatomy of biological motion perception in humans , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[68]  Asif A Ghazanfar,et al.  Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys , 2008, The Journal of Neuroscience.

[69]  C. Rorden,et al.  Stereotaxic display of brain lesions. , 2000, Behavioural neurology.

[70]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[71]  Alfonso Caramazza,et al.  The logic of neuropsychological research and the problem of patient classification in aphasia , 1984, Brain and Language.

[72]  A. Cowey,et al.  Sensitivity to eye gaze in prosopagnosic patients and monkeys with superior temporal sulcus ablation , 1990, Neuropsychologia.

[73]  Yaoda Xu,et al.  Distinctive Neural Mechanisms Supporting Visual Object Individuation and Identification , 2009, Journal of Cognitive Neuroscience.