Temporal factors affecting somatosensory–auditory interactions in speech processing

Speech perception is known to rely on both auditory and visual information. However, sound-specific somatosensory input has been shown also to influence speech perceptual processing (Ito et al., 2009). In the present study, we addressed further the relationship between somatosensory information and speech perceptual processing by addressing the hypothesis that the temporal relationship between orofacial movement and sound processing contributes to somatosensory–auditory interaction in speech perception. We examined the changes in event-related potentials (ERPs) in response to multisensory synchronous (simultaneous) and asynchronous (90 ms lag and lead) somatosensory and auditory stimulation compared to individual unisensory auditory and somatosensory stimulation alone. We used a robotic device to apply facial skin somatosensory deformations that were similar in timing and duration to those experienced in speech production. Following synchronous multisensory stimulation the amplitude of the ERP was reliably different from the two unisensory potentials. More importantly, the magnitude of the ERP difference varied as a function of the relative timing of the somatosensory–auditory stimulation. Event-related activity change due to stimulus timing was seen between 160 and 220 ms following somatosensory onset, mostly around the parietal area. The results demonstrate a dynamic modulation of somatosensory–auditory convergence and suggest the contribution of somatosensory information for speech processing process is dependent on the specific temporal order of sensory inputs in speech production.

[1]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[2]  N. Geschwind.  The Organization of Language and the Brain: Language disorders after brain damage help in elucidating the neural basis of verbal behavior , 1970 .

[3]  Marco Iacoboni,et al.  The Essential Role of Premotor Cortex in Speech Perception , 2007, Current Biology.

[4]  Takayuki Ito,et al.  Speech sounds alter facial skin sensation. , 2012, Journal of neurophysiology.

[5]  Louis Goldstein,et al.  Bridging planning and execution: Temporal planning of syllables , 2012, J. Phonetics.

[6]  Jianxun Zhou,et al.  Somatosensory influence on the cochlear nucleus and beyond , 2006, Hearing Research.

[7]  Riitta Hari,et al.  Touch activates human auditory cortex , 2006, NeuroImage.

[8]  C. Fowler,et al.  Listening with eye and hand: Cross-modal contributions to speech perception. , 1991 .

[9]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[10]  N. Logothetis,et al.  Integration of Touch and Sound in Auditory Cortex , 2005, Neuron.

[11]  A. Boothroyd,et al.  Cortical Evoked Response To Acoustic Change Within A Syllable , 1998, Ear and hearing.

[12]  R. Diehl,et al.  Speech Perception , 2004, Annual review of psychology.

[13]  Mikko Sams,et al.  Viewing speech modulates activity in the left SI mouth cortex , 2005, NeuroImage.

[14]  K. Watkins,et al.  Motor Representations of Articulators Contribute to Categorical Perception of Speech Sounds , 2009, The Journal of Neuroscience.

[15]  K. Munhall,et al.  Compensation following real-time manipulation of formants in isolated vowels. , 2006, The Journal of the Acoustical Society of America.

[16]  K. Tremblay,et al.  Speech Evoked Potentials: From the Laboratory to the Clinic , 2008, Ear and hearing.

[17]  Feng Rong,et al.  Sensorimotor Integration in Speech Processing: Computational Basis and Neural Organization , 2011, Neuron.

[18]  John J. Foxe,et al.  Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. , 2005, Cerebral cortex.

[19]  B. Gick,et al.  Aero-tactile integration in speech perception , 2009, Nature.

[20]  Ankoor S. Shah,et al.  Auditory Cortical Neurons Respond to Somatosensory Stimulation , 2003, The Journal of Neuroscience.

[21]  Joanne L. Miller,et al.  Speech Perception , 1990, Springer Handbook of Auditory Research.

[22]  B. Stein,et al.  Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[24]  T. Picton,et al.  The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. , 1987, Psychophysiology.

[25]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[26]  David J. Ostry,et al.  Somatosensory function in speech perception , 2009, Proceedings of the National Academy of Sciences.

[27]  G. Rizzolatti,et al.  Speech listening specifically modulates the excitability of tongue muscles: a TMS study , 2002, The European journal of neuroscience.

[28]  Rainer Goebel,et al.  The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. , 2006, Cerebral cortex.

[29]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[30]  G. V. Simpson,et al.  Flow of activation from V1 to frontal cortex in humans , 2001, Experimental Brain Research.

[31]  D. Lehmann,et al.  Reference-free identification of components of checkerboard-evoked multichannel potential fields. , 1980, Electroencephalography and clinical neurophysiology.

[32]  Michael Petrides,et al.  Distinct Parietal and Temporal Connectivity Profiles of Ventrolateral Frontal Areas Involved in Language Production , 2013, The Journal of Neuroscience.

[33]  C. Lammertmann,et al.  Magnetoencephalographic Correlates of Audiotactile Interaction , 2002, NeuroImage.

[34]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[35]  N. Geschwind.  The organization of language and the brain. , 1970, Science.

[36]  A M Liberman,et al.  Perception of the speech code. , 1967, Psychological review.

[37]  David J Ostry,et al.  Somatosensory contribution to motor learning due to facial skin deformation. , 2010, Journal of neurophysiology.

[38]  L. Fadiga,et al.  The Motor Somatotopy of Speech Perception , 2009, Current Biology.

[39]  T. Paus,et al.  Seeing and hearing speech excites the motor system involved in speech production , 2003, Neuropsychologia.

[40]  Denis Brunet,et al.  Topographic ERP Analyses: A Step-by-Step Tutorial Review , 2008, Brain Topography.

[41]  E. Toppila,et al.  The effect of different noise types on the speech and non-speech elicited mismatch negativity , 2005, Hearing Research.

[42]  C. Spence,et al.  Audiotactile interactions in temporal perception , 2011, Psychonomic bulletin & review.

[43]  M. Iacoboni,et al.  Listening to speech activates motor areas involved in speech production , 2004, Nature Neuroscience.

[44]  Jean Vroomen,et al.  Visual Anticipatory Information Modulates Multisensory Interactions of Artificial Audiovisual Stimuli , 2010, Journal of Cognitive Neuroscience.

[45]  John J. Foxe,et al.  Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. , 2002, Journal of neurophysiology.

[46]  Mikko Sams,et al.  Perceiving identical sounds as speech or non-speech modulates activity in the left posterior superior temporal sulcus , 2006, NeuroImage.

[47]  Michael S. Beauchamp,et al.  Touch, sound and vision in human superior temporal sulcus , 2008, NeuroImage.

[48]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[49]  C. Fowler,et al.  Listening with eye and hand: cross-modal contributions to speech perception. , 1991, Journal of experimental psychology. Human perception and performance.

[50]  V. Gracco,et al.  Regional heterogeneity in the processing and the production of speech in the human planum temporale , 2013, Cortex.

[51]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[52]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[53]  Alexis R. Johns,et al.  Left lateralized enhancement of orofacial somatosensory processing due to speech sounds. , 2013, Journal of speech, language, and hearing research : JSLHR.

[54]  J. Vroomen,et al.  Perception of intersensory synchrony: A tutorial review , 2010, Attention, perception & psychophysics.

[55]  Z. Wang,et al.  The influence of temporal asynchrony on multisensory integration in the processing of asynchronous audio-visual stimuli of real-world events: an event-related potential study , 2011, Neuroscience.

[56]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[57]  J. Schwartz,et al.  The Perception-for-Action-Control Theory (PACT): A perceptuo-motor theory of speech perception , 2012, Journal of Neurolinguistics.

[58]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[59]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[60]  Michael Pilling.  Auditory event-related potentials (ERPs) in audiovisual speech perception. , 2009, Journal of speech, language, and hearing research : JSLHR.