Multisensory speech perception without the left superior temporal sulcus

Converging evidence suggests that the left superior temporal sulcus (STS) is a critical site for multisensory integration of auditory and visual information during speech perception. We report a patient, SJ, who suffered a stroke that damaged the left tempo-parietal area, resulting in mild anomic aphasia. Structural MRI showed complete destruction of the left middle and posterior STS, as well as damage to adjacent areas in the temporal and parietal lobes. Surprisingly, SJ demonstrated preserved multisensory integration measured with two independent tests. First, she perceived the McGurk effect, an illusion that requires integration of auditory and visual speech. Second, her perception of morphed audiovisual speech with ambiguous auditory or visual information was significantly influenced by the opposing modality. To understand the neural basis for this preserved multisensory integration, blood-oxygen level dependent functional magnetic resonance imaging (BOLD fMRI) was used to examine brain responses to audiovisual speech in SJ and 23 healthy age-matched controls. In controls, bilateral STS activity was observed. In SJ, no activity was observed in the damaged left STS but in the right STS, more cortex was active in SJ than in any of the normal controls. Further, the amplitude of the BOLD response in right STS response to McGurk stimuli was significantly greater in SJ than in controls. The simplest explanation of these results is a reorganization of SJ's cortical language networks such that the right STS now subserves multisensory integration of speech.

[1]  D. C. Howell,et al.  Comparing an Individual's Test Score Against Norms Derived from Small Samples , 1998 .

[2]  Michael S Beauchamp,et al.  Neural Correlates of Interindividual Differences in Children's Audiovisual Speech Perception , 2011, The Journal of Neuroscience.

[3]  S. Scott,et al.  The neuroanatomical and functional organization of speech perception , 2003, Trends in Neurosciences.

[4]  N. Dronkers,et al.  Lesion analysis of the brain areas involved in language comprehension , 2004, Cognition.

[5]  B. Argall,et al.  Simplified intersubject averaging on the cortical surface using SUMA , 2006, Human brain mapping.

[6]  M. Kinsbourne.  The minor cerebral hemisphere as a source of aphasic speech. , 1971, Archives of neurology.

[7]  Anders M. Dale,et al.  Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature , 2010, NeuroImage.

[8]  Abraham Z. Snyder,et al.  Word Retrieval Learning Modulates Right Frontal Cortex in Patients with Left Frontal Damage , 2002, Neuron.

[9]  M. Seghier,et al.  An anatomical signature for literacy , 2009, Nature.

[10]  David Howard,et al.  Neuropsychological studies of auditory-visual fusion illusions. Four case studies and their implications , 1990, Neuropsychologia.

[11]  Nadia Bolognini,et al.  Audiovisual Integration in Patients with Visual Deficit , 2005, Journal of Cognitive Neuroscience.

[12]  C. Coelho,et al.  Auditory-visual speech perception in an adult with aphasia , 2004, Brain injury.

[13]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[14]  M Corbetta,et al.  Preserved speech abilities and compensation following prefrontal damage. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[15]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[16]  E Altenmüller,et al.  Language processing in aphasia: changes in lateralization patterns during recovery reflect cerebral plasticity in adults. , 1997, Electroencephalography and clinical neurophysiology.

[17]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[18]  Russell A. Poldrack,et al.  Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses , 2012, NeuroImage.

[19]  A. Thiel,et al.  Role of the Contralateral Inferior Frontal Gyrus in Recovery of Language Function in Poststroke Aphasia: A Combined Repetitive Transcranial Magnetic Stimulation and Positron Emission Tomography Study , 2005, Stroke.

[20]  Gregory Hickok,et al.  The role of Broca’s area in speech perception: Evidence from aphasia revisited , 2011, Brain and Language.

[21]  Antoinette T. Gesi,et al.  Bimodal speech perception: an examination across languages , 1993 .

[22]  Dries J. W. Froyen,et al.  Exploring the Role of Low Level Visual Processing in Letter–Speech Sound Integration: A Visual MMN Study , 2010, Frontiers in Integrative Neuroscience.

[23]  Franco Lepore,et al.  A role for the inferior colliculus in multisensory speech integration , 2006, Neuroreport.

[24]  Wolfram Ziegler,et al.  The influence of visual and auditory information on the perception of speech and non‐speech oral movements in patients with left hemisphere lesions , 2009, Clinical linguistics & phonetics.

[25]  Cécile Colin,et al.  Top-down and bottom-up modulation of audiovisual integration in speech , 2005 .

[26]  A M Dale,et al.  Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[27]  B. Stein,et al.  The Merging of the Senses , 1993 .

[28]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[29]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[30]  P F Seitz,et al.  The use of visible speech cues for improving auditory detection of spoken sentences. , 2000, The Journal of the Acoustical Society of America.

[31]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[32]  Audrey R. Nath,et al.  Dynamic Changes in Superior Temporal Sulcus Connectivity during Perception of Noisy Audiovisual Speech , 2011, The Journal of Neuroscience.

[33]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[34]  H. Pashler,et al.  Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1 , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[35]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[36]  Ryan A. Stevenson,et al.  Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition , 2009, NeuroImage.

[37]  S. Scott,et al.  Speech comprehension aided by multiple modalities: Behavioural and neural interactions , 2012, Neuropsychologia.

[38]  Audrey R. Nath,et al.  fMRI-Guided Transcranial Magnetic Stimulation Reveals That the Superior Temporal Sulcus Is a Cortical Locus of the McGurk Effect , 2010, The Journal of Neuroscience.

[39]  Ferath Kherif,et al.  The Main Sources of Intersubject Variability in Neuronal Activation for Reading Aloud , 2009, Journal of Cognitive Neuroscience.

[40]  David B. Pisoni,et al.  Discrete neural substrates underlie complementary audiovisual speech integration processes , 2011, NeuroImage.

[41]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[42]  Michael S. Beauchamp,et al.  A new method for improving functional-to-structural MRI alignment using local Pearson correlation , 2009, NeuroImage.

[43]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[44]  John C. Gore,et al.  ROC Analysis of Statistical Methods Used in Functional MRI: Individual Subjects , 1999, NeuroImage.

[45]  D. V. van Essen,et al.  Corticocortical connections of visual, sensorimotor, and multimodal processing areas in the parietal lobe of the macaque monkey , 2000, The Journal of comparative neurology.

[46]  R. Goebel,et al.  Integration of Letters and Speech Sounds in the Human Brain , 2004, Neuron.

[47]  Rainer Goebel,et al.  Task‐irrelevant visual letters interact with the processing of speech sounds in heteromodal and unimodal cortex , 2008, The European journal of neuroscience.

[48]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[49]  Michael S. Beauchamp,et al.  A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion , 2012, NeuroImage.

[50]  Nancy Kanwisher,et al.  Divide and conquer: A defense of functional localizers , 2006, NeuroImage.

[51]  Rainer Goebel,et al.  fMR-adaptation indicates selectivity to audiovisual content congruency in distributed clusters in human superior temporal cortex , 2010, BMC Neuroscience.

[52]  B. Seltzer,et al.  Overlapping and nonoverlapping cortical projections to cortex of the superior temporal sulcus in the rhesus monkey: Double anterograde tracer studies , 1996 .

[53]  Nikos Makris,et al.  Automatically parcellating the human cerebral cortex. , 2004, Cerebral cortex.

[54]  J Mazziotta,et al.  A probabilistic atlas and reference system for the human brain: International Consortium for Brain Mapping (ICBM). , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[55]  M. Kinsbourne.  The Minor Cerebral Hemisphere , 1971 .

[56]  R. C. Oldfield.  The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[57]  Roy H. Hamilton,et al.  The right hemisphere is not unitary in its role in aphasia recovery , 2012, Cortex.

[58]  Karl J. Friston,et al.  A critique of functional localisers , 2006, NeuroImage.

[59]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[60]  K P George,et al.  Cortical language activation in stroke patients recovering from aphasia with functional MRI. , 1999, Stroke.