Adaptation Aftereffects in Vocal Emotion Perception Elicited by Expressive Faces and Voices

The perception of emotions is often suggested to be multimodal in nature, and bimodal as compared to unimodal (auditory or visual) presentation of emotional stimuli can lead to superior emotion recognition. In previous studies, contrastive aftereffects in emotion perception caused by perceptual adaptation have been shown for faces and for auditory affective vocalization, when adaptors were of the same modality. By contrast, crossmodal aftereffects in the perception of emotional vocalizations have not been demonstrated yet. In three experiments we investigated the influence of emotional voice as well as dynamic facial video adaptors on the perception of emotion-ambiguous voices morphed on an angry-to-happy continuum. Contrastive aftereffects were found for unimodal (voice) adaptation conditions, in that test voices were perceived as happier after adaptation to angry voices, and vice versa. Bimodal (voice + dynamic face) adaptors tended to elicit larger contrastive aftereffects. Importantly, crossmodal (dynamic face) adaptors also elicited substantial aftereffects in male, but not in female participants. Our results (1) support the idea of contrastive processing of emotions (2), show for the first time crossmodal adaptation effects under certain conditions, consistent with the idea that emotion processing is multimodal in nature, and (3) suggest gender differences in the sensory integration of facial and vocal emotional stimuli.

[1]  Andrew W. Young,et al.  Neuropsychology of fear and loathing , 2001, Nature Reviews Neuroscience.

[2]  Frans A. J. Verstraten,et al.  The motion aftereffect , 1998, Trends in Cognitive Sciences.

[3]  Tania Singer,et al.  University of Zurich Zurich Open Repository and Archive I Feel How You Feel but Not Always: the Empathic Brain and Its I Feel How You Feel but Not Always: the Empathic Brain and Its Modulation , 2009 .

[4]  Christopher J. Fox,et al.  What is adapted in face adaptation? The neural representations of expression in the human visual system , 2007, Brain Research.

[5]  Agneta Herlitz,et al.  Higher face recognition ability in girls: Magnified by own-sex and own-ethnicity bias , 2006, Memory.

[6]  G. Rhodes,et al.  The dynamics of visual adaptation to faces. , 2005, Proceedings. Biological sciences.

[7]  G. Kovács,et al.  Young without plastic surgery: Perceptual adaptation to the age of female and male faces , 2010, Vision Research.

[8]  C. Spence,et al.  Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli , 2007, Perception & psychophysics.

[9]  S. Campanella,et al.  Integrating face and voice in person perception , 2007, Trends in Cognitive Sciences.

[10]  K. Scherer,et al.  Acoustic profiles in vocal emotion expression. , 1996, Journal of personality and social psychology.

[11]  Hideki Kawahara,et al.  Auditory Adaptation in Voice Perception , 2008, Current Biology.

[12]  J. Montepare,et al.  The Use of Body Movements and Gestures as Cues to Emotions in Younger and Older Adults , 1999 .

[13]  M. Yücel,et al.  Sex differences in the neural correlates of emotion: Evidence from neuroimaging , 2011, Biological Psychology.

[14]  M. Kret,et al.  A review on sex differences in processing emotional signals , 2012, Neuropsychologia.

[15]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[16]  Perceptual Auditory Aftereffects on Voice Identity Using Brief Vowel Stimuli , 2012, PloS one.

[17]  A. Friederici,et al.  Sex differentiates the role of emotional prosody during word processing. , 2002, Brain research. Cognitive brain research.

[18]  Hideki Kawahara,et al.  Development of exploratory research tools based on TANDEM-STRAIGHT , 2009 .

[19]  Annett Schirmer,et al.  On the role of attention for the processing of emotions in speech: sex differences revisited. , 2005, Brain research. Cognitive brain research.

[20]  Michael J Brammer,et al.  Crossmodal identification , 1998, Trends in Cognitive Sciences.

[21]  Hideki Kawahara,et al.  Voice aftereffects of adaptation to speaker identity , 2010, Hearing Research.

[22]  Stefan R Schweinberger,et al.  What drives social in-group biases in face recognition memory? ERP evidence from the own-gender bias. , 2014, Social cognitive and affective neuroscience.

[23]  H. Huynh,et al.  Estimation of the Box Correction for Degrees of Freedom from Sample Data in Randomized Block and Split-Plot Designs , 1976 .

[24]  K. Scherer,et al.  Emotion expression in body action and posture. , 2012, Emotion.

[25]  Stefan R. Schweinberger,et al.  You are only as old as you sound: Auditory aftereffects in vocal age perception , 2011, Hearing Research.

[26]  Michael Erb,et al.  Impact of voice on emotional judgment of faces: An event‐related fMRI study , 2006, Human brain mapping.

[27]  A. Young,et al.  MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus , 2009, Proceedings of the National Academy of Sciences.

[28]  Glyn W. Humphreys,et al.  Expression is computed separately from facial identity, and it is computed separately for moving and static faces: Neuropsychological evidence , 1993, Neuropsychologia.

[29]  F. Gosselin,et al.  Audio-visual integration of emotion expression , 2008, Brain Research.

[30]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[31]  Hideki Kawahara,et al.  Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[33]  Nicu Sebe,et al.  Emotion Recognition Based on Joint Visual and Audio Cues , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[34]  E. Kreta,et al.  A review on sex differences in processing emotional signals , 2012 .

[35]  B. Murphy,et al.  Adaptation to natural facial categories , 2022 .

[36]  P. Belin,et al.  Auditory adaptation in vocal affect perception , 2010, Cognition.

[37]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[38]  Hideki Kawahara,et al.  In the ear of the beholder: neural correlates of adaptation to voice gender , 2009, The European journal of neuroscience.

[39]  Sophie K. Scott,et al.  Impaired auditory recognition of fear and anger following bilateral amygdala lesions , 1997, Nature.

[40]  R. Marken.  Methods in experimental psychology , 1981 .

[41]  A. Young,et al.  Understanding person perception. , 2011, British journal of psychology.

[42]  Verena G. Skuk,et al.  Gender differences in familiar voice identification , 2013, Hearing Research.

[43]  Armin Bruderlin,et al.  Perceiving affect from arm movement , 2001, Cognition.

[44]  R. Dolan,et al.  Crossmodal binding of fear in voice and face , 2001, Proceedings of the National Academy of Sciences of the United States of America.