Crossmodal interaction of facial and vocal person identity information: An event-related potential study

Hearing a voice and seeing a face are essential parts of person identification and social interaction. It has been suggested that both types of information do not only interact at late processing stages but rather interact at the level of perceptual encoding (<200 ms). The present study analysed when visual and auditory representations of person identity modulate the processing of voices. In unimodal trials, two successive voices (S1-S2) of the same or of two different speakers were presented. In the crossmodal condition, the S1 consisted of the face of the same or a different person with respect to the following voice stimulus. Participants had to decide whether the voice probe (S2) was from an elderly or a young person. Reaction times to the S2 were shorter when these stimuli were person-congruent, both in the uni- and crossmodal conditions. ERPs recorded to the person-incongruent as compared to the person-congruent trials (S2) were enhanced at early (100-140 ms) and later processing stages (270-530 ms) in the crossmodal condition. A similar later negative ERP effect (270-530 ms) was found in the unimodal condition as well. These results suggest that identity information conveyed by a face is capable to modulate the sensory processing of voice stimuli.

[1]  A. Giraud,et al.  Implicit Multisensory Associations Influence Voice Recognition , 2006, PLoS biology.

[2]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[3]  Robert Oostenveld,et al.  The five percent electrode system for high-resolution EEG and ERP measurements , 2001, Clinical Neurophysiology.

[4]  Y. Arimatsu,et al.  Foreign gene expression in an organotypic culture of cortical anlage after in vivo electroporation. , 1999, Neuroreport.

[5]  John J. Foxe,et al.  Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. , 2004, Cerebral cortex.

[6]  O. Bertrand,et al.  Visual Activation and Audiovisual Interactions in the Auditory Cortex during Speech Perception: Intracranial Recordings in Humans , 2008, The Journal of Neuroscience.

[7]  S. Schweinberger,et al.  Neuropsychological Impairments in the Recognition of Faces, Voices, and Personal Names , 2000, Brain and Cognition.

[8]  K. Grill-Spector,et al.  Repetition and the brain: neural models of stimulus-specific effects , 2006, Trends in Cognitive Sciences.

[9]  M. Kutas,et al.  The Search for Common Sense: An Electrophysiological Study of the Comprehension of Words and Pictures in Reading , 1996, Journal of Cognitive Neuroscience.

[10]  R. Zatorre,et al.  Voice-selective areas in human auditory cortex , 2000, Nature.

[11]  Stefan J Kiebel,et al.  How the Human Brain Recognizes Speech in the Context of Changing Speakers , 2010, The Journal of Neuroscience.

[12]  M. Crommelinck,et al.  When audition alters vision: an event-related potential study of the cross-modal interactions between faces and voices , 2004, Neuroscience Letters.

[13]  J. Kaiser,et al.  Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas , 2007, The Journal of Neuroscience.

[14]  Eckart Altenmüller,et al.  Neural processing of vocal emotion and identity , 2009, Brain and Cognition.

[15]  Karl J. Friston,et al.  The effect of prior visual information on recognition of speech and sounds. , 2008, Cerebral cortex.

[16]  Takashi Tsukiura,et al.  Orbitofrontal and hippocampal contributions to memory for face–name associations: The rewarding power of a smile , 2008, Neuropsychologia.

[17]  J. Haxby,et al.  Distinct representations of eye gaze and identity in the distributed human neural system for face perception , 2000, Nature Neuroscience.

[18]  K. Fujii,et al.  Visualization for the analysis of fluid motion , 2005, J. Vis..

[19]  Stefan R Schweinberger,et al.  Human brain potential correlates of voice priming and voice recognition , 2001, Neuropsychologia.

[20]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[21]  Christoph M. Michel,et al.  Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging , 2004, NeuroImage.

[22]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[23]  Hideki Kawahara,et al.  In the ear of the beholder: neural correlates of adaptation to voice gender , 2009, The European journal of neuroscience.

[24]  P. Holcomb,et al.  An electrophysiological investigation of semantic priming with pictures of real objects. , 1999, Psychophysiology.

[25]  Anne-Lise Giraud,et al.  Distinct functional substrates along the right superior temporal sulcus for the processing of voices , 2004, NeuroImage.

[26]  T. Stanford,et al.  Multisensory integration: current issues from the perspective of the single neuron , 2008, Nature Reviews Neuroscience.

[27]  A. Young,et al.  Understanding face recognition. , 1986, British journal of psychology.

[28]  T. Paine.  Common Sense , 1995 .

[29]  M. Rugg,et al.  Event-related potentials and the semantic matching of faces , 1989, Neuropsychologia.

[30]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[31]  A. Damasio,et al.  Intact recognition of facial expression, gender, and age in patients with impaired recognition of face identity , 1988, Neurology.

[32]  Werner Sommer,et al.  Repetition priming and associative priming of face recognition: Evidence from event-related potentials. , 1995 .

[33]  R. Näätänen.  Attention and brain function , 1992 .

[34]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[35]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[36]  S. Campanella,et al.  Integrating face and voice in person perception , 2007, Trends in Cognitive Sciences.

[37]  Fang Fang,et al.  Perceptual grouping and inverse fMRI activity patterns in human visual cortex. , 2008, Journal of vision.

[38]  Yoko Mano,et al.  Dissociable Roles of the Anterior Temporal Regions in Successful Encoding of Memory for Person Identity Information , 2010, Journal of Cognitive Neuroscience.

[39]  D. Maurer,et al.  Neural correlates of processing facial identity based on features versus their spacing , 2007, Neuropsychologia.

[40]  Pascal Belin,et al.  Electrophysiological evidence for an early processing of human voices , 2009, BMC Neuroscience.

[41]  P. Belin,et al.  Thinking the voice: neural correlates of voice perception , 2004, Trends in Cognitive Sciences.

[42]  P. Rämä,et al.  Separate neural processes for retrieval of voice identity and word content in working memory , 2009, Brain Research.

[43]  Patrik Vuilleumier,et al.  Research report T processing of emotional facial expression is gated by spatial attention: evidence from event-related brain potentials , 2003 .

[44]  R. Zatorre,et al.  Human temporal-lobe response to vocal sounds. , 2002, Brain research. Cognitive brain research.

[45]  J Grigor,et al.  The effect of odour priming on long latency visual evoked potentials of matching and mismatching objects. , 1999, Chemical senses.

[46]  L M Ward,et al.  Multisensory integration and crossmodal attention effects in the human brain. , 2001, Science.

[47]  Jonathan Grainger,et al.  An electrophysiological study of cross-modal repetition priming. , 2005, Psychophysiology.

[48]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[49]  R. Zatorre,et al.  Adaptation to speaker's voice in right anterior temporal lobe , 2003, Neuroreport.

[50]  H. Huynh,et al.  Estimation of the Box Correction for Degrees of Freedom from Sample Data in Randomized Block and Split-Plot Designs , 1976 .

[51]  K. Zilles,et al.  The neural correlates of person familiarity. A functional magnetic resonance imaging study with clinical implications. , 2001, Brain : a journal of neurology.

[52]  Andreas Kleinschmidt,et al.  Interaction of Face and Voice Areas during Speaker Recognition , 2005, Journal of Cognitive Neuroscience.

[53]  Dylan M. Jones,et al.  Intra- and inter-modal repetition priming of familiar faces and voices. , 1997, British journal of psychology.

[54]  A. Cowey,et al.  The role of the 'face-cell' area in the discrimination and recognition of faces by monkeys. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[55]  Deborah J. Hellawell,et al.  Facial expression processing after amygdalotomy , 1996, Neuropsychologia.

[56]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[57]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[58]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[59]  Werner Sommer,et al.  Memory systems for structural and semantic knowledge of faces and buildings , 2006, Brain Research.

[60]  M. Tarr,et al.  Expertise Training with Novel Objects Leads to Left-Lateralized Facelike Electrophysiological Responses , 2002, Psychological science.

[61]  Christoph Kayser,et al.  Do early sensory cortices integrate cross-modal information? , 2007, Brain Structure and Function.

[62]  Brigitte Röder,et al.  Multisensory processing in the redundant-target effect: A behavioral and event-related potential study , 2005, Perception & psychophysics.

[63]  A. Cowey,et al.  Sensitivity to eye gaze in prosopagnosic patients and monkeys with superior temporal sulcus ablation , 1990, Neuropsychologia.

[64]  J. Stekelenburg,et al.  Naso-temporal asymmetry of the N170 for processing faces in normal viewers but not in developmental prosopagnosia , 2005, Neuroscience Letters.

[65]  C. Rennie,et al.  Decrement of the N1 auditory event-related potential with stimulus repetition: habituation vs. refractoriness. , 1998, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[66]  P. Rämä,et al.  N400 during recognition of voice identity and vocal affect , 2009, Neuroreport.

[67]  P. Belin,et al.  Electrophysiological markers of voice familiarity , 2006, The European journal of neuroscience.

[68]  M. Eimer.  The face‐specific N170 component reflects late stages in the structural encoding of faces , 2000, Neuroreport.

[69]  C. Petten,et al.  Conceptual relationships between spoken words and environmental sounds: Event-related brain potential measures , 1995, Neuropsychologia.

[70]  Olivier Bertrand,et al.  Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex , 2009, Hearing Research.

[71]  A. Burton,et al.  Human brain potential correlates of repetition priming in face and name recognition , 2002, Neuropsychologia.

[72]  B. Rossion,et al.  Right N170 modulation in a face discrimination task: an account for categorical perception of familiar faces. , 2000, Psychophysiology.

[73]  A. Amedi,et al.  Functional imaging of human crossmodal identification and object recognition , 2005, Experimental Brain Research.

[74]  Robert Oostenveld,et al.  Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming , 2008, NeuroImage.

[75]  Jean Vroomen,et al.  Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events , 2007, Journal of Cognitive Neuroscience.

[76]  Asif A Ghazanfar,et al.  Interactions between the Superior Temporal Sulcus and Auditory Cortex Mediate Dynamic Face/Voice Integration in Rhesus Monkeys , 2008, The Journal of Neuroscience.

[77]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[78]  A. Kleinschmidt,et al.  Modulation of neural responses to speech by directing attention to voices or verbal content. , 2003, Brain research. Cognitive brain research.

[79]  M. Murray,et al.  The role of multisensory memories in unisensory object discrimination. , 2005, Brain research. Cognitive brain research.

[80]  L. Benevento,et al.  Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey , 1977, Experimental Neurology.

[81]  Takashi Tsukiura,et al.  Dissociable roles of the bilateral anterior temporal lobe in face−name associations: An event-related fMRI study , 2006, NeuroImage.

[82]  Pierre Maurage,et al.  The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study , 2011, NeuroImage.

[83]  T. Allison,et al.  Temporal Cortex Activation in Humans Viewing Eye and Mouth Movements , 1998, The Journal of Neuroscience.

[84]  Sönke Johannes,et al.  Brain potentials reveal the timing of face identity and expression judgments , 1998, Neuroscience Research.

[85]  S. Bentin,et al.  Processing specificity for human voice stimuli: electrophysiological evidence , 2001, Neuroreport.

[86]  D. Schacter,et al.  Priming and the Brain , 1998, Neuron.