Audiovisual integration during speech comprehension: An fMRI study comparing ROI‐based and whole brain analyses

Visual information (lip movements) significantly contributes to speech comprehension raising the question for the neural implementation of audiovisual (AV) integration during speech processing. To replicate and extend earlier neuroimaging findings, we compared two different analysis approaches in a slow event‐related fMRI study of healthy native speakers of German who were exposed to AV speech stimuli (disyllabic nouns) with audio and visual signals being either congruent or incongruent. First, data was subjected to whole brain general linear model analysis after transformation of all individual data sets into standard space. Second, a region of interest (ROI) approach based on individual anatomy was used with ROI defined in areas identified previously as being important for AV processing. Standard space analysis revealed a widespread cortical network including the posterior part of the left superior temporal sulcus, Broca's region and its right hemispheric counterpart showing increased activity for incongruent stimuli. The ROI approach allowed to identify differences in activity between Brodmann areas 44 and 45, within Broca's area for incongruent stimulation, and also allowed to study activity of subdivisions of superior temporal regions. The complementary strengths and weaknesses of the two analysis approaches are discussed. Hum Brain Mapp, 2009. © 2008 Wiley‐Liss, Inc.

[1]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[2]  M. Botvinick,et al.  Anterior cingulate cortex, error detection, and the online monitoring of performance. , 1998, Science.

[3]  E. C. Cherry.  Some Experiments on the Recognition of Speech, with One and with Two Ears , 1953 .

[4]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[5]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[6]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[7]  M. Corballis.  From mouth to hand: Gesture, speech, and the evolution of right-handedness , 2003, Behavioral and Brain Sciences.

[8]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[9]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[10]  P. Morosan,et al.  Human Primary Auditory Cortex: Cytoarchitectonic Subdivisions and Mapping into a Spatial Reference System , 2001, NeuroImage.

[11]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[12]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[13]  M. Arbib,et al.  Language within our grasp , 1998, Trends in Neurosciences.

[14]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[15]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[16]  Mikko Sams,et al.  Processing of audiovisual speech in Broca's area , 2005, NeuroImage.

[17]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[18]  M. Frens,et al.  Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements , 1995, Perception & psychophysics.

[19]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[20]  Jeffery A. Jones,et al.  Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect , 2003, Neuroreport.

[21]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[22]  J. Mazziotta,et al.  The essential role of Broca's area in imitation , 2003, The European journal of neuroscience.

[23]  E Macaluso,et al.  Spatial and temporal factors during processing of audiovisual speech: a PET study , 2004, NeuroImage.

[24]  P. Hagoort.  On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[25]  M HERSHENSON,et al.  Reaction time as a measure of intersensory facilitation. , 1962, Journal of experimental psychology.

[26]  Jeremy I. Skipper,et al.  Seeing Voices : How Cortical Areas Supporting Speech Production Mediate Audiovisual Speech Perception , 2007 .

[27]  T. Crow,et al.  Right hemisphere language functions and schizophrenia: the forgotten hemisphere? , 2005, Brain : a journal of neurology.

[28]  Mikko Sams,et al.  Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: An fMRI study at 3 T , 2006, NeuroImage.

[29]  Jelliffe.  Vergleichende Lokalisationslehre der Grosshirnrinde , 1910 .

[30]  K. Amunts,et al.  Broca's region: from action to language. , 2005, Physiology.

[31]  D E Callan,et al.  Multimodal contribution to speech perception revealed by independent component analysis: a single-sweep EEG case study. , 2001, Brain research. Cognitive brain research.

[32]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[33]  T. Paus,et al.  Regional differences in the effects of task difficulty and motor output on blood flow response in the human anterior cingulate cortex: a review of 107 PET activation studies , 1998, Neuroreport.

[34]  Jean-Francois Mangin,et al.  Sulcal pattern and morphology of the superior temporal sulcus , 2004, NeuroImage.

[35]  M. Arbib.  From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics , 2005, Behavioral and Brain Sciences.

[36]  D. Pandya,et al.  Anatomy of the auditory cortex. , 1995, Revue neurologique.

[37]  M. Hallett,et al.  Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection , 2001, The Journal of Neuroscience.

[38]  M. Sams,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003, Brain research. Cognitive brain research.

[39]  Friedemann Pulvermüller,et al.  Brain mechanisms linking language and action , 2005, Nature Reviews Neuroscience.

[40]  Rainer Goebel,et al.  Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single‐subject to cortically aligned group general linear model analysis and self‐organizing group independent component analysis , 2006, Human brain mapping.

[41]  R. Töpper,et al.  Motor cortex hand area and speech: implications for the development of language , 2003, Neuropsychologia.

[42]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[43]  Alan C. Evans,et al.  Morphology, morphometry and probability mapping of the pars opercularis of the inferior frontal gyrus: an in vivo MRI analysis , 1999, The European journal of neuroscience.

[44]  Lynne E. Bernstein,et al.  Spatiotemporal dynamics of audiovisual speech processing , 2008, NeuroImage.

[45]  S. Bookheimer.  Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[46]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[47]  J Driver,et al.  Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. , 2000, Journal of neurophysiology.

[48]  A. Schleicher,et al.  Broca's region revisited: Cytoarchitecture and intersubject variability , 1999, The Journal of comparative neurology.

[49]  K. R. Ridderinkhof,et al.  The Role of the Medial Frontal Cortex in Cognitive Control , 2004, Science.

[50]  C Tempelmann,et al.  Electrodynamic headphones and woofers for application in magnetic resonance imaging scanners. , 1998, Medical physics.

[51]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[52]  W. Lutzenberger,et al.  Sequential audiovisual interactions during speech perception: A whole-head MEG study , 2007, Neuropsychologia.

[53]  Thomas E. Nichols,et al.  Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate , 2002, NeuroImage.

[54]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[55]  R. H. Baayen,et al.  The CELEX Lexical Database (CD-ROM) , 1996 .

[56]  Steven L. Small,et al.  Listening to talking faces: motor cortical activation during speech perception , 2005, NeuroImage.

[57]  L. Morrell,et al.  Temporal characteristics of sensory interaction in choice reaction times. , 1968, Journal of experimental psychology.

[58]  Alan C. Evans,et al.  Interhemispheric anatomical differences in human primary auditory cortex: probabilistic mapping and volume measurement from magnetic resonance scans. , 1996, Cerebral cortex.

[59]  J. Rothwell,et al.  Speech‐induced changes in corticospinal excitability , 1996, Annals of neurology.

[60]  Lawrence G. McDade,et al.  Behavioral Indices of Multisensory Integration: Orientation to Visual Cues is Affected by Auditory Stimuli , 1989, Journal of Cognitive Neuroscience.

[61]  Arthur W. Toga,et al.  Temporal and Topographical Characterization of Language Cortices Using Intraoperative Optical Intrinsic Signals , 2000, NeuroImage.

[62]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[63]  J. Mazziotta,et al.  Functional segregation within pars opercularis of the inferior frontal gyrus: evidence from fMRI studies of imitation and action observation. , 2005, Cerebral cortex.

[64]  André Brechmann,et al.  Sound-level-dependent representation of frequency modulations in human auditory cortex: a low-noise fMRI study. , 2002, Journal of neurophysiology.

[65]  A. Galaburda,et al.  Cytoarchitectonic organization of the human auditory cortex , 1980, The Journal of comparative neurology.

[66]  T. Paus.  Primate anterior cingulate cortex: Where motor control, drive and cognition interface , 2001, Nature Reviews Neuroscience.

[67]  Y. Sugita,et al.  Auditory-visual speech perception examined by fMRI and PET , 2003, Neuroscience Research.

[68]  M. Botvinick,et al.  The Contribution of the Anterior Cingulate Cortex to Executive Processes in Cognition , 1999, Reviews in the neurosciences.

[69]  Chris I. Baker,et al.  Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions , 2005, Journal of Cognitive Neuroscience.

[70]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[71]  Brian H Scott,et al.  Cortical mechanisms in hearing , 2003, Current Opinion in Neurobiology.

[72]  M. Corbetta,et al.  Selective and divided attention during visual discriminations of shape, color, and speed: functional anatomy by positron emission tomography , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[73]  S. Bookheimer,et al.  Dissociating Neural Mechanisms of Temporal Sequencing and Processing Phonemes , 2003, Neuron.

[74]  P. Morosan,et al.  Probabilistic Mapping and Volume Measurement of Human Primary Auditory Cortex , 2001, NeuroImage.

[75]  Peggy Tausche,et al.  A novel approach to study audiovisual integration in speech perception: Localizer fMRI and sparse sampling , 2008, Brain Research.

[76]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[77]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[78]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[79]  M. Botvinick,et al.  Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[80]  E. C. Cmm,et al.  on the Recognition of Speech, with , 2008 .

[81]  G. A. Calvert,et al.  Auditory-visual processing represented in the human superior temporal gyrus , 2007, Neuroscience.