Crossmodal Interactions in Audiovisual Emotion Processing Veronika I. Studies of Audiovisual Integration Have Predominantly Assessed the Neural Correlates of Audiovisual Speech Perception (beauchamp Unimodal Emotional Processing in Turn Has Been Extensively Studied, with a High Proportion of Studies

Emotion in daily life is often expressed in a multimodal fashion. Consequently emotional information from one modality can influence processing in another. In a previous fMRI study we assessed the neural correlates of audio-visual integration and found that activity in the left amygdala is significantly attenuated when a neutral stimulus is paired with an emotional one compared to conditions where emotional stimuli were present in both channels. Here we used dynamic causal modelling to investigate the effective connectivity in the neuronal network underlying this emotion presence congruence effect. Our results provided strong evidence in favor of a model family, differing only in the interhemispheric interactions. All winning models share a connection from the bilateral fusiform gyrus (FFG) into the left amygdala and a non-linear modulatory influence of bilateral posterior superior temporal sulcus (pSTS) on these connections. This result indicates that the pSTS not only integrates multi-modal information from visual and auditory regions (as reflected in our model by significant feed-forward connections) but also gates the influence of the sensory information on the left amygdala, leading to attenuation of amygdala activity when a neutral stimulus is integrated. Moreover, we found a significant lateralization of the FFG due to stronger driving input by the stimuli (faces) into the right hemisphere, whereas such lateralization was not present for sound-driven input into the superior temporal gyrus. In summary, our data provides further evidence for a rightward lateralization of the FFG and in particular for a key role of the pSTS in the integration and gating of audio-visual emotional information.

[1]  Simon B. Eickhoff,et al.  Dynamic intra- and interhemispheric interactions during unilateral and bilateral hand movements assessed with fMRI and DCM , 2008, NeuroImage.

[2]  Frank Schneider,et al.  Incongruence effects in crossmodal emotional integration , 2011, NeuroImage.

[3]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[4]  Audrey R. Nath,et al.  fMRI-Guided Transcranial Magnetic Stimulation Reveals That the Superior Temporal Sulcus Is a Cortical Locus of the McGurk Effect , 2010, The Journal of Neuroscience.

[5]  W. Grodd,et al.  Association of trait emotional intelligence and individual fMRI‐activation patterns during the perception of social signals from voice and face , 2009, Human brain mapping.

[6]  R. Zatorre,et al.  Voice-selective areas in human auditory cortex , 2000, Nature.

[7]  M. Erb,et al.  Effects of prosodic emotional intensity on activation of associative auditory cortex , 2006, Neuroreport.

[8]  A. McDonald.  Cortical pathways to the mammalian amygdala , 1998, Progress in Neurobiology.

[9]  C. Price,et al.  The role of the posterior superior temporal sulcus in audiovisual processing. , 2008, Cerebral cortex.

[10]  David C. Alsop,et al.  Brain Activation during Facial Emotion Processing , 2002, NeuroImage.

[11]  R. Schultz,et al.  Superior temporal activation in response to dynamic audio-visual emotional cues , 2009, Brain and Cognition.

[12]  G. Rhodes,et al.  Is the Fusiform Face Area Specialized for Faces, Individuation, or Expert Individuation? , 2004, Journal of Cognitive Neuroscience.

[13]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[14]  Robert T. Knight,et al.  Superior Temporal SulcusIt's My Area: Or Is It? , 2008, Journal of Cognitive Neuroscience.

[15]  J. Dien.  A tale of two recognition systems: Implications of the fusiform face area and the visual word form area for lateralized object recognition models , 2009, Neuropsychologia.

[16]  Ross M. VanDerKlok,et al.  Multisensory perception of action in posterior temporal and parietal cortices , 2011, Neuropsychologia.

[17]  Robert T. Schultz,et al.  Bidirectional communication between amygdala and fusiform gyrus during facial recognition , 2011, NeuroImage.

[18]  D. N. Pandya,et al.  Laminar Termination Patterns of Thalamic, Callosal, and Association Afferents in the Primary Auditory Area of the Rhesus Monkey , 1993, Experimental Neurology.

[19]  Y. Sugita,et al.  Auditory-visual speech perception examined by fMRI and PET , 2003, Neuroscience Research.

[20]  Simon B. Eickhoff,et al.  A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data , 2005, NeuroImage.

[21]  H. Ackermann,et al.  Semiotic aspects of human nonverbal vocalizations: a functional imaging study , 2007, Neuroreport.

[22]  G. Pourtois,et al.  Perception of Facial Expressions and Voices and of their Combination in the Human Brain , 2005, Cortex.

[23]  Michael Erb,et al.  Audiovisual integration of emotional signals in voice and face: An event-related fMRI study , 2007, NeuroImage.

[24]  Simon B. Eickhoff,et al.  Dynamic causal modeling of cortical activity from the acute to the chronic stage after stroke , 2011, NeuroImage.

[25]  Gereon R Fink,et al.  Noradrenergic enhancement improves motor network connectivity in stroke patients , 2011, Annals of neurology.

[26]  Wolfgang Grodd,et al.  Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice- and face-sensitive regions in the superior temporal sulcus , 2009, Neuropsychologia.

[27]  Ewald Moser,et al.  Amygdala activation and facial expressions: Explicit emotion discrimination versus implicit emotion processing , 2007, Neuropsychologia.

[28]  Kevin A. Pelphrey,et al.  Face processing without awareness in the right fusiform gyrus , 2007, Neuropsychologia.

[29]  Pascal Belin,et al.  Amygdala responses to nonlinguistic emotional vocalizations , 2007, NeuroImage.

[30]  Simon B. Eickhoff,et al.  Assignment of functional activations to probabilistic cytoarchitectonic areas revisited , 2007, NeuroImage.

[31]  T. Ban,et al.  Commissural afferents to the cortex surrounding the posterior part of the superior temporal sulcus in the monkey , 1984, Neuroscience Letters.

[32]  D. V. van Essen,et al.  The pattern of interhemispheric connections and its relationship to extrastriate visual areas in the macaque monkey , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[33]  A. Young,et al.  MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus , 2009, Proceedings of the National Academy of Sciences.

[34]  Research report A comparison of bound and unbound audio-visual information processing in the human cerebral cortex , 2002 .

[35]  L. Abbott,et al.  Synaptic computation , 2004, Nature.

[36]  H Scheich,et al.  Auditory perception of laughing and crying activates human amygdala regardless of attentional state. , 2001, Brain research. Cognitive brain research.

[37]  F. Lin,et al.  Primary and multisensory cortical activity is correlated with audiovisual percepts , 2009, Human brain mapping.

[38]  R. Dolan,et al.  Crossmodal binding of fear in voice and face , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[39]  K. Amunts,et al.  A systems perspective on the effective connectivity of overt speech production , 2009, NeuroImage.

[40]  Michael S. Beauchamp,et al.  A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion , 2012, NeuroImage.

[41]  Karl J. Friston,et al.  Bayesian model selection for group studies , 2009, NeuroImage.

[42]  R. C. Oldfield.  The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[43]  Qingyang Li,et al.  Emotional perception: Meta-analyses of face and natural scene processing , 2011, NeuroImage.

[44]  A. Ishai,et al.  Effective connectivity within the distributed cortical network for face perception. , 2007, Cerebral cortex.

[45]  Karl J. Friston,et al.  Nonlinear Dynamic Causal Models for Fmri Nonlinear Dynamic Causal Models for Fmri Nonlinear Dynamic Causal Models for Fmri , 2022 .

[46]  J. Kaas,et al.  Callosal connections of the parabelt auditory cortex in macaque monkeys , 1999, The European journal of neuroscience.

[47]  A. Young,et al.  Neural responses to facial and vocal expressions of fear and disgust , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[48]  Karl J. Friston,et al.  A unified statistical approach for determining significant signals in images of cerebral activation , 1996, Human brain mapping.

[49]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[50]  R. Adolphs,et al.  Annals of the New York Academy of Sciences What Does the Amygdala Contribute to Social Cognition? , 2022 .

[51]  宁北芳,et al.  疟原虫var基因转换速率变化导致抗原变异[英]／Paul H, Robert P, Christodoulou Z, et al//Proc Natl Acad Sci U S A , 2005 .

[52]  Masao Yukie,et al.  Connections between the amygdala and auditory cortical areas in the macaque monkey , 2002, Neuroscience Research.

[53]  Guido Gainotti,et al.  What the study of voice recognition in normal subjects and brain-damaged patients tells us about models of familiar people recognition , 2011, Neuropsychologia.

[54]  N. Kanwisher,et al.  The fusiform face area subserves face perception, not generic within-category identification , 2004, Nature Neuroscience.

[55]  Wolfgang Grodd,et al.  Cerebral processing of emotional prosody—influence of acoustic parameters and arousal , 2008, NeuroImage.

[56]  Simon B. Eickhoff,et al.  The state of tranquility: Subjective perception is shaped by contextual modulation of auditory connectivity , 2010, NeuroImage.

[57]  D Yves von Cramon,et al.  Distinct fMRI responses to laughter, speech, and sounds along the human peri-sylvian cortex. , 2005, Brain research. Cognitive brain research.

[58]  S. Clarke,et al.  Occipital cortex in man: Organization of callosal connections, related myelo‐ and cytoarchitecture, and putative boundaries of functional visual areas , 1990, The Journal of comparative neurology.

[59]  Gilles Pourtois,et al.  Investigating audiovisual integration of emotional signals in the human brain. , 2006, Progress in brain research.

[60]  Karl J. Friston,et al.  Dissociating Reading Processes on the Basis of Neuronal Interactions , 2005, Journal of Cognitive Neuroscience.

[61]  Simon B. Eickhoff,et al.  Dynamic interactions in the fronto-parietal network during a manual stimulus–response compatibility task , 2011, NeuroImage.

[62]  P. Belin,et al.  Thinking the voice: neural correlates of voice perception , 2004, Trends in Cognitive Sciences.

[63]  T. Kircher,et al.  Supramodal Representation of Emotions , 2011, The Journal of Neuroscience.

[64]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[65]  L. Tyler,et al.  Binding crossmodal object features in perirhinal cortex. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[66]  P. McGuire,et al.  Laterality effect on emotional faces processing: ALE meta-analysis of evidence , 2009, Neuroscience Letters.

[67]  Karl J. Friston,et al.  Ten simple rules for dynamic causal modeling , 2010, NeuroImage.

[68]  Conny F. Schmidt,et al.  Face perception is mediated by a distributed cortical network , 2005, Brain Research Bulletin.

[69]  Simon B. Eickhoff,et al.  Testing anatomically specified hypotheses in functional imaging using cytoarchitectonic maps , 2006, NeuroImage.

[70]  M. Wallace,et al.  A revised view of sensory cortical parcellation , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[71]  N. Kanwisher,et al.  The fusiform face area: a cortical region specialized for the perception of faces , 2006, Philosophical Transactions of the Royal Society B: Biological Sciences.