fMRI-Guided Transcranial Magnetic Stimulation Reveals That the Superior Temporal Sulcus Is a Cortical Locus of the McGurk Effect

A compelling example of auditory–visual multisensory integration is the McGurk effect, in which an auditory syllable is perceived very differently depending on whether it is accompanied by a visual movie of a speaker pronouncing the same syllable or a different, incongruent syllable. Anatomical and physiological studies in human and nonhuman primates have suggested that the superior temporal sulcus (STS) is involved in auditory–visual integration for both speech and nonspeech stimuli. We hypothesized that the STS plays a critical role in the creation of the McGurk percept. Because the location of multisensory integration in the STS varies from subject to subject, the location of auditory–visual speech processing in the STS was first identified in each subject with fMRI. Then, activity in this region of the STS was disrupted with single-pulse transcranial magnetic stimulation (TMS) as subjects rated their percept of McGurk and non-McGurk stimuli. Across three experiments, TMS of the STS significantly reduced the likelihood of the McGurk percept but did not interfere with perception of non-McGurk stimuli. TMS of the STS was effective at disrupting the McGurk effect only in a narrow temporal window from 100 ms before auditory syllable onset to 100 ms after onset, and TMS of a control location did not influence perception of McGurk or control stimuli. These results demonstrate that the STS plays a critical role in the McGurk effect and auditory–visual integration of speech.

[1]  R. Raizada,et al.  Quantifying the adequacy of neural representations for a cross-language phonetic discrimination task: prediction of individual differences. , 2010, Cerebral cortex.

[2]  Christoph Kayser,et al.  Spatial Organization of Multisensory Responses in Temporal Association Cortex , 2009, The Journal of Neuroscience.

[3]  Rainer Goebel,et al.  Optimizing Functional Accuracy of TMS in Cognitive Studies: A Comparison of Methods , 2009, Journal of Cognitive Neuroscience.

[4]  L. Bernstein,et al.  Quantified acoustic–optical speech signal incongruity identifies cortical sites of audiovisual speech processing , 2008, Brain Research.

[5]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[6]  C. Price,et al.  The role of the posterior superior temporal sulcus in audiovisual processing. , 2008, Cerebral cortex.

[7]  Michael S. Beauchamp,et al.  Touch, sound and vision in human superior temporal sulcus , 2008, NeuroImage.

[8]  T. Paus,et al.  Transcranial magnetic stimulation and the challenge of coil placement: A comparison of conventional and stereotaxic neuronavigational strategies , 2008, Human brain mapping.

[9]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[10]  N. Barbaro,et al.  Spatiotemporal Dynamics of Word Processing in the Human Brain , 2007, Front. Neurosci..

[11]  Daniela Balslev,et al.  Inter-individual variability in optimal current direction for transcranial magnetic stimulation of the motor cortex , 2007, Journal of Neuroscience Methods.

[12]  Aina Puce,et al.  Neural responses elicited to face motion and vocalization pairings , 2007, Neuropsychologia.

[13]  Rainer Goebel,et al.  The temporal characteristics of motion processing in hMT/V5+: Combining fMRI and neuronavigated TMS , 2006, NeuroImage.

[14]  Jean-Francois Mangin,et al.  Modulation of language areas with functional MR image-guided magnetic stimulation , 2006, NeuroImage.

[15]  B. Argall,et al.  Simplified intersubject averaging on the cortical surface using SUMA , 2006, Human brain mapping.

[16]  Tracy R. Henderson,et al.  Simple metric for scaling motor threshold based on scalp-cortex distance: application to studies using transcranial magnetic stimulation. , 2005, Journal of neurophysiology.

[17]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[18]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[19]  Chris I. Baker,et al.  Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions , 2005, Journal of Cognitive Neuroscience.

[20]  Michael S. Beauchamp,et al.  Statistical criteria in fMRI studies of multisensory integration , 2005, Neuroinformatics.

[21]  David C. Van Essen,et al.  Surface-based approaches to spatial localization and registration in primate cerebral cortex , 2004, NeuroImage.

[22]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[23]  Jeffery A. Jones,et al.  Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information , 2004, Journal of Cognitive Neuroscience.

[24]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[25]  Tony Ro,et al.  Visual Enhancing of Tactile Perception in the Posterior Parietal Cortex , 2004, Journal of Cognitive Neuroscience.

[26]  D. V. van Essen,et al.  Surface-based approaches to spatial localization and registration in primate cerebral cortex. , 2004, NeuroImage.

[27]  Nikos Makris,et al.  Automatically parcellating the human cerebral cortex. , 2004, Cerebral cortex.

[28]  Y. Sugita,et al.  Auditory-visual speech perception examined by fMRI and PET , 2003, Neuroscience Research.

[29]  Jeffery A. Jones,et al.  Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect , 2003, Neuroreport.

[30]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[31]  J Mazziotta,et al.  A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[32]  J MacDonald,et al.  Hearing by Eye: How Much Spatial Degradation can Be Tolerated? , 2000, Perception.

[33]  Alan Cowey,et al.  Transcranial magnetic stimulation and cognitive neuroscience , 2000, Nature Reviews Neuroscience.

[34]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[35]  J. Rothwell,et al.  Transcranial magnetic stimulation in cognitive neuroscience – virtual lesion, chronometry, and functional connectivity , 2000, Current Opinion in Neurobiology.

[36]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[37]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[38]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[39]  積山 薫.  McGurk Effect and Incompatibility : A Cross-Language study on Auditory-Visual Speech Perception , 1994 .

[40]  Michael Wilson,et al.  MRC psycholinguistic database: Machine-usable dictionary, version 2.00 , 1988 .

[41]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.