Investigating audiovisual integration of emotional signals in the human brain.

Humans can communicate their emotional state via facial expression and affective prosody. This chapter reviews behavioural, neuroanatomical, electrophysiological and neuroimaging studies pertaining to audiovisual integration of emotional communicative signals. Particular emphasis will be given to neuroimaging studies using positron emission tomography (PET) or functional magnetic resonance imaging (fMRI). Conjunction analyses, interaction analyses, correlation analyses between haemodynamic responses and behavioural effects and connectivity analyses have been employed to analyse neuroimaging data. There is no general agreement as to which of these approaches can be considered "optimal" to classify brain regions as multisensory. We argue that these approaches provide complementing information as they assess different aspects of multisensory integration of emotional information. Assets and drawbacks of the different analysis types are discussed and demonstrated on the basis of one fMRI data set.

[1]  Karl J. Friston,et al.  Multisubject fMRI Studies and Conjunction Analyses , 1999, NeuroImage.

[2]  R. Turner,et al.  Form and motion coherence activate independent, but not dorsal/ventral segregated, networks in the human brain , 2000, Current Biology.

[3]  B. Stein,et al.  The Merging of the Senses , 1993 .

[4]  M. Mesulam,et al.  From sensation to cognition. , 1998, Brain : a journal of neurology.

[5]  Maurizio Corbetta,et al.  The human brain is intrinsically organized into dynamic, anticorrelated functional networks. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[6]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[7]  M White.  Representation of facial expressions of emotion. , 1999, The American journal of psychology.

[8]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[9]  J Driver,et al.  Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. , 2000, Journal of neurophysiology.

[10]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[11]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[12]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[13]  Karl J. Friston,et al.  Modeling regional and psychophysiologic interactions in fMRI: the importance of hemodynamic deconvolution , 2003, NeuroImage.

[14]  Deepak N. Pandya,et al.  Further observations on corticofrontal connections in the rhesus monkey , 1976, Brain Research.

[15]  Michael Erb,et al.  Impact of voice on emotional judgment of faces: An event‐related fMRI study , 2006, Human brain mapping.

[16]  Keiji Tanaka,et al.  Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. , 1988, Journal of neurophysiology.

[17]  E. Schröger,et al.  Speeded responses to audiovisual signal changes result from bimodal integration. , 1998, Psychophysiology.

[18]  M. Wallace,et al.  Integration of multiple sensory modalities in cat cortex , 2004, Experimental Brain Research.

[19]  K. Zilles,et al.  Crossmodal Processing of Object Features in Human Anterior Intraparietal Cortex An fMRI Study Implies Equivalencies between Humans and Monkeys , 2002, Neuron.

[20]  S. Rauch,et al.  Response and Habituation of the Human Amygdala during Visual Processing of Facial Expression , 1996, Neuron.

[21]  C. K. Peck.  Visual-auditory interactions in cat superior colliculus: their role in the control of gaze , 1987, Brain Research.

[22]  Jyrki Tuomainen,et al.  The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses , 1999, Neuroscience Letters.

[23]  B. Gordon,et al.  Receptive fields in deep layers of cat superior colliculus. , 1973, Journal of neurophysiology.

[24]  Keith J. Holyoak,et al.  Structure and Functions of the Human Prefrontal Cortex , 1996 .

[25]  A. McDonald.  Cortical pathways to the mammalian amygdala , 1998, Progress in Neurobiology.

[26]  Karl J. Friston,et al.  Cognitive Conjunction: A New Approach to Brain Activation Experiments , 1997, NeuroImage.

[27]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[28]  P. Goldman-Rakic.  Architecture of the Prefrontal Cortex and the Central Executive , 1995, Annals of the New York Academy of Sciences.

[29]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[30]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[31]  M M Mesulam,et al.  Thalamic connections of the insula in the rhesus monkey and comments on the paralimbic connectivity of the medial pulvinar nucleus , 1984, The Journal of comparative neurology.

[32]  Frequency-dependent inhibition to tones in neurons of cat insular cortex (AIV) , 1978, Brain Research.

[33]  Chris I. Baker,et al.  Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions , 2005, Journal of Cognitive Neuroscience.

[34]  R. Dolan,et al.  Crossmodal binding of fear in voice and face , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[35]  M Mishkin,et al.  Organization of the amygdalopetal projections from modality‐specific cortical association areas in the monkey , 1980, The Journal of comparative neurology.

[36]  P. Ekman.  Pictures of Facial Affect , 1976 .

[37]  T. P. S. Powell,et al.  The organization of the connections between the cortex and the claustrum in the monkey , 1982, Brain Research.

[38]  B. Stein,et al.  Interactions among converging sensory inputs in the superior colliculus. , 1983, Science.

[39]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[40]  G. Pourtois,et al.  Perception of Facial Expressions and Voices and of their Combination in the Human Brain , 2005, Cortex.

[41]  Andreas Kleinschmidt,et al.  Interaction of Face and Voice Areas during Speaker Recognition , 2005, Journal of Cognitive Neuroscience.

[42]  T. Powell,et al.  An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. , 1970, Brain : a journal of neurology.

[43]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[44]  R. Desimone,et al.  Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. , 1981, Journal of neurophysiology.

[45]  A. Damasio.  Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition , 1989, Cognition.

[46]  Karl J. Friston,et al.  A neuromodulatory role for the human amygdala in processing emotional facial expressions. , 1998, Brain : a journal of neurology.

[47]  D. Perrett,et al.  A differential neural response in the human amygdala to fearful and happy facial expressions , 1996, Nature.

[48]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[49]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[50]  J. Vroomen,et al.  The perception of emotions by ear and by eye , 2000 .

[51]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[52]  Jon Driver,et al.  Is cross-modal integration of emotional expressions independent of attentional resources? , 2001, Cognitive, affective & behavioral neuroscience.

[53]  M. Mesulam,et al.  Insula of the old world monkey. III: Efferent cortical output and comments on function , 1982, The Journal of comparative neurology.

[54]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[55]  John Patrick Aggleton,et al.  The Amygdala : a functional analysis , 2000 .

[56]  B. Rossion,et al.  The time‐course of intermodal binding between seeing and hearing affective information , 2000, Neuroreport.

[57]  Sophie K. Scott,et al.  Impaired auditory recognition of fear and anger following bilateral amygdala lesions , 1997, Nature.

[58]  D. Pandya,et al.  Afferent cortical connections and architectonics of the superior temporal sulcus and surrounding cortex in the rhesus monkey , 1978, Brain Research.

[59]  W. Fries.  Cortical projections to the superior colliculus in the macaque monkey: A retrograde study using horseradish peroxidase , 1984, The Journal of comparative neurology.

[60]  M Mishkin,et al.  Amygdalectomy impairs crossmodal association in monkeys. , 1985, Science.

[61]  Gilles Pourtois,et al.  Facial expressions modulate the time course of long latency auditory brain potentials. , 2002, Brain research. Cognitive brain research.

[62]  Karl J. Friston,et al.  Conjunction revisited , 2005, NeuroImage.

[63]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[64]  Jeffery A. Jones,et al.  Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect , 2003, Neuroreport.

[65]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[66]  C. Spence,et al.  Multisensory perception: Beyond modularity and convergence , 2000, Current Biology.

[67]  M. Escobar,et al.  In vivo long-term potentiation in the insular cortex: NMDA receptor dependence , 1998, Brain Research.

[68]  M. Wallace,et al.  Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. , 1993, Journal of neurophysiology.

[69]  J. Gore,et al.  A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. , 2002, Brain research. Cognitive brain research.

[70]  Karl J. Friston,et al.  Nonlinear Responses in fMRI: The Balloon Model, Volterra Kernels, and Other Hemodynamics , 2000, NeuroImage.

[71]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[72]  L. Benevento,et al.  Auditory-visual interaction in single units in the orbito-insular cortex of the cat. , 1969, Electroencephalography and clinical neurophysiology.

[73]  D. Collins,et al.  Automatic 3D Intersubject Registration of MR Volumetric Data in Standardized Talairach Space , 1994, Journal of computer assisted tomography.

[74]  R. Adolphs,et al.  Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala , 1994, Nature.

[75]  A. Young,et al.  Neural responses to facial and vocal expressions of fear and disgust , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[76]  Karl J. Friston,et al.  Psychophysiological and Modulatory Interactions in Neuroimaging , 1997, NeuroImage.

[77]  Karl J. Friston,et al.  A unified statistical approach for determining significant signals in images of cerebral activation , 1996, Human brain mapping.

[78]  D. Massaro,et al.  Perceptual recognition of facial affect: Cross-cultural comparisons , 1996, Memory & cognition.

[79]  Jesper Andersson,et al.  Valid conjunction inference with the minimum statistic , 2005, NeuroImage.

[80]  M T Wallace,et al.  Comparisons of cross-modality integration in midbrain and cortex. , 1996, Progress in brain research.

[81]  E. DeYoe,et al.  A comparison of visual and auditory motion processing in human cerebral cortex. , 2000, Cerebral cortex.

[82]  Karl J. Friston,et al.  Nonlinear event‐related responses in fMRI , 1998, Magnetic resonance in medicine.

[83]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[84]  Michael S. Beauchamp,et al.  Statistical criteria in fMRI studies of multisensory integration , 2005, Neuroinformatics.