Multisensory and modality specific processing of visual speech in different regions of the premotor cortex

Behavioral and neuroimaging studies have demonstrated that brain regions involved with speech production also support speech perception, especially under degraded conditions. The premotor cortex (PMC) has been shown to be active during both observation and execution of action (“Mirror System” properties), and may facilitate speech perception by mapping unimodal and multimodal sensory features onto articulatory speech gestures. For this functional magnetic resonance imaging (fMRI) study, participants identified vowels produced by a speaker in audio-visual (saw the speaker's articulating face and heard her voice), visual only (only saw the speaker's articulating face), and audio only (only heard the speaker's voice) conditions with varying audio signal-to-noise ratios in order to determine the regions of the PMC involved with multisensory and modality specific processing of visual speech gestures. The task was designed so that identification could be made with a high level of accuracy from visual only stimuli to control for task difficulty and differences in intelligibility. The results of the functional magnetic resonance imaging (fMRI) analysis for visual only and audio-visual conditions showed overlapping activity in inferior frontal gyrus and PMC. The left ventral inferior premotor cortex (PMvi) showed properties of multimodal (audio-visual) enhancement with a degraded auditory signal. The left inferior parietal lobule and right cerebellum also showed these properties. The left ventral superior and dorsal premotor cortex (PMvs/PMd) did not show this multisensory enhancement effect, but there was greater activity for the visual only over audio-visual conditions in these areas. The results suggest that the inferior regions of the ventral premotor cortex are involved with integrating multisensory information, whereas, more superior and dorsal regions of the PMC are involved with mapping unimodal (in this case visual) sensory features of the speech signal with articulatory speech gestures.

[1]  Takashi Hanakawa,et al.  Song and speech: Brain regions involved with perception and covert production , 2006, NeuroImage.

[2]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[3]  Edward E. Smith,et al.  The Role of Parietal Cortex in Verbal Working Memory , 1998, The Journal of Neuroscience.

[4]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[5]  Marco Iacoboni,et al.  The Essential Role of Premotor Cortex in Speech Perception , 2007, Current Biology.

[6]  J. Gore,et al.  A comparison of bound and unbound audio-visual information processing in the human cerebral cortex. , 2002, Brain research. Cognitive brain research.

[7]  Masa-aki Sato,et al.  Premotor cortex mediates perceptual performance , 2010, NeuroImage.

[8]  L. Braida,et al.  Evaluating the articulation index for auditory-visual input. , 1987, The Journal of the Acoustical Society of America.

[9]  P. Strick,et al.  Cerebellar output channels. , 1997, International review of neurobiology.

[10]  Lila Davachi,et al.  When Keeping in Mind Supports Later Bringing to Mind: Neural Markers of Phonological Rehearsal Predict Subsequent Remembering , 2001, Journal of Cognitive Neuroscience.

[11]  Jeremy I. Skipper,et al.  Speech-associated gestures, Broca’s area, and the human mirror system , 2007, Brain and Language.

[12]  Jeffery A. Jones,et al.  Phonetic perceptual identification by native- and second-language speakers differentially activates brain regions involved with acoustic phonetic processing and those involved with articulatory–auditory/orosensory internal models , 2004, NeuroImage.

[13]  Y. Sugita,et al.  Auditory-visual speech perception examined by fMRI and PET , 2003, Neuroscience Research.

[14]  Hiroshi Imamizu,et al.  Human cerebellar activity reflecting an acquired internal model of a new tool , 2000, Nature.

[15]  A. Liberman,et al.  On the relation of speech to language , 2000, Trends in Cognitive Sciences.

[16]  R. Hari,et al.  Viewing Lip Forms Cortical Dynamics , 2002, Neuron.

[17]  D E Callan,et al.  Multimodal contribution to speech perception revealed by independent component analysis: a single-sweep EEG case study. , 2001, Brain research. Cognitive brain research.

[18]  A. Kok,et al.  Neuroelectrical signs of selective attention to color in boys with attention-deficit hyperactivity disorder. , 2001, Brain research. Cognitive brain research.

[19]  Mikko Sams,et al.  Enhanced early-latency electromagnetic activity in the left premotor cortex is associated with successful phonetic categorization , 2012, NeuroImage.

[20]  A M Liberman,et al.  Perception of the speech code. , 1967, Psychological review.

[21]  Frank H Guenther,et al.  The DIVA model: A neural theory of speech acquisition and production , 2011, Language and cognitive processes.

[22]  L. Bernstein,et al.  Visual speech perception without primary auditory cortex activation , 2002, Neuroreport.

[23]  D E Callan,et al.  Single-sweep EEG analysis of neural processes underlying perception and production of vowels. , 2000, Brain research. Cognitive brain research.

[24]  Angela D. Friederici,et al.  Pathways to language: fiber tracts in the human brain , 2009, Trends in Cognitive Sciences.

[25]  P. McGuire,et al.  Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower-face acts (gurning). , 2001, Brain research. Cognitive brain research.

[26]  N. A. Borghese,et al.  A functional-anatomical model for lipreading. , 2003, Journal of neurophysiology.

[27]  Mitsuo Kawato,et al.  Internal models for motor control and trajectory planning , 1999, Current Opinion in Neurobiology.

[28]  Jeremy I. Skipper,et al.  Seeing Voices : How Cortical Areas Supporting Speech Production Mediate Audiovisual Speech Perception , 2007 .

[29]  H. Otzenberger,et al.  Visemic processing in audiovisual discrimination of natural speech: A simultaneous fMRI–EEG study , 2012, Neuropsychologia.

[30]  D. Callan,et al.  Cerebellar control of speech and song , 2013 .

[31]  R. Campbell,et al.  Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech , 2003, Journal of Cognitive Neuroscience.

[32]  Vincent L. Gracco,et al.  A mediating role of the premotor cortex in phoneme segmentation , 2009, Brain and Language.

[33]  B. Stein,et al.  The Merging of the Senses , 1993 .

[34]  Jeffery A. Jones,et al.  Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information , 2004, Journal of Cognitive Neuroscience.

[35]  Steven L. Small,et al.  A Network Model of Observation and Imitation of Speech , 2012, Front. Psychology.

[36]  A. Liberman,et al.  The motor theory of speech perception revised , 1985, Cognition.

[37]  Marco Iacoboni,et al.  Beyond a Single Area: Motor Control and Language Within a Neural Architecture Encompassing Broca's Area , 2006, Cortex.

[38]  Manabu Honda,et al.  Cross-modal binding and activated attentional networks during audio-visual speech integration: a functional MRI study. , 2005, Cerebral cortex.

[39]  D. Pandya,et al.  The cerebrocerebellar system. , 1997, International review of neurobiology.

[40]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[41]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[42]  Mikko Sams,et al.  Processing of audiovisual speech in Broca's area , 2005, NeuroImage.

[43]  Demis Basso,et al.  Sensory‐motor brain network connectivity for speech comprehension , 2009, Human brain mapping.

[44]  M. Wallace,et al.  Integration of multiple sensory modalities in cat cortex , 2004, Experimental Brain Research.

[45]  Louis D. Braida,et al.  Evaluating the articulation index for auditory-visual input. , 1987, The Journal of the Acoustical Society of America.

[46]  Marco Iacoboni,et al.  Neural responses to non-native phonemes varying in producibility: Evidence for the sensorimotor nature of speech perception , 2006, NeuroImage.

[47]  K. Amunts,et al.  Broca's region: from action to language. , 2005, Physiology.

[48]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[49]  D. Poeppel,et al.  Towards a functional neuroanatomy of speech perception , 2000, Trends in Cognitive Sciences.

[50]  Marco Iacoboni,et al.  The role of premotor cortex in speech perception: Evidence from fMRI and rTMS , 2008, Journal of Physiology-Paris.

[51]  Frank H. Guenther,et al.  A neural theory of speech acquisition and production , 2012, Journal of Neurolinguistics.

[52]  P. Denes.  On the Motor Theory of Speech Perception , 1965 .

[53]  D. Poeppel,et al.  Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language , 2004, Cognition.

[54]  Ken W. Grant,et al.  Erratum: ‘‘Evaluating the articulation index for auditory–visual input’’ [J. Acoust. Soc. Am. 89, 2952–2960 (1991)] , 1991 .

[55]  J. Schwartz,et al.  The Perception-for-Action-Control Theory (PACT): A perceptuo-motor theory of speech perception , 2012, Journal of Neurolinguistics.

[56]  Keiichi Tajima,et al.  Neural processes involved with perception of non-native durational contrasts , 2006, Neuroreport.

[57]  Steven L. Small,et al.  Listening to talking faces: motor cortical activation during speech perception , 2005, NeuroImage.

[58]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[59]  D. Poeppel,et al.  Speech perception at the interface of neurobiology and linguistics , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[60]  Josef P. Rauschecker,et al.  An expanded role for the dorsal auditory pathway in sensorimotor control and integration , 2011, Hearing Research.

[61]  Mitsuo Kawato,et al.  Speech and song: The role of the cerebellum , 2008, The Cerebellum.

[62]  Richard H Wilson,et al.  Auditory Test No . 6 in multi-talker babble : A preliminary report , 2002 .

[63]  Raymond D. Kent,et al.  An auditory-feedback-based neural network model of speech production that is robust to developmental changes in the size and shape of the articulatory system. , 2000, Journal of speech, language, and hearing research : JSLHR.

[64]  M. Iacoboni,et al.  Listening to speech activates motor areas involved in speech production , 2004, Nature Neuroscience.