Electrophysiological (EEG, sEEG, MEG) evidence for multiple audiovisual interactions in the human auditory cortex

In this review, we examine the contribution of human electrophysiological studies (EEG, sEEG and MEG) to the study of visual influence on processing in the auditory cortex. Focusing mainly on studies performed by our group, we critically review the evidence showing (1) that visual information can both activate and modulate the activity of the auditory cortex at relatively early stages (mainly at the processing stage of the auditory N1 wave) in response to both speech and non-speech sounds and (2) that visual information can be included in the representation of both speech and non-speech sounds in auditory sensory memory. We describe an important conceptual tool in the study of audiovisual interaction (the additive model) and show the importance of considering the spatial distribution of electrophysiological data when interpreting EEG results. Review of these studies points to the probable role of sensory, attentional and task-related factors in modulating audiovisual interactions in the auditory cortex.

[1]  John J. Foxe,et al.  Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: a high-density electrical mapping study. , 2005, Cerebral cortex.

[2]  J Miller,et al.  Channel interaction and the redundant-targets effect in bimodal divided attention. , 1991, Journal of experimental psychology. Human perception and performance.

[3]  John J. Foxe,et al.  Multisensory processing of naturalistic objects in motion: A high-density electrical mapping and source estimation study , 2007, NeuroImage.

[4]  A. Dittmann-Balçar,et al.  Attention-dependent allocation of auditory processing resources as measured by mismatch negativity. , 1999, Neuroreport.

[5]  I. Winkler,et al.  Preattentive representation of feature conjunctions for concurrent spatially distributed auditory objects. , 2005, Brain research. Cognitive brain research.

[6]  John J. Foxe,et al.  Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion , 2007, Neuropsychologia.

[7]  C Alain,et al.  Effects of visual attentional load on auditory processing , 2000, Neuroreport.

[8]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[9]  Robert Oostenveld,et al.  Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming , 2008, NeuroImage.

[10]  Anne Caclin,et al.  Audiovisual Events in Sensory Memory , 2007 .

[11]  I. Czigler.  Visual mismatch negativity: Violation of nonattended environmental regularities , 2007 .

[12]  Zhong-Lin Lu,et al.  Modification of sudden onset auditory ERP by involuntary attention to visual stimuli. , 2002, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[13]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[14]  P. Deltenre,et al.  Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonants , 2004, Clinical Neurophysiology.

[15]  Mikko Sams,et al.  Seeing speech affects acoustic information processing in the human brainstem , 2005, Experimental Brain Research.

[16]  A. Muller-Gass,et al.  The effect of visual task difficulty and attentional direction on the detection of acoustic change as indexed by the Mismatch Negativity , 2006, Brain Research.

[17]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[18]  M. Sams,et al.  Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[19]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[20]  J Pernier,et al.  Neurophysiological mechanisms of auditory selective attention in humans. , 2000, Frontiers in bioscience : a journal and virtual library.

[21]  J. Pernier,et al.  Early auditory-visual interactions in human cortex during nonredundant target identification. , 2002, Brain research. Cognitive brain research.

[22]  Peter Ullsperger,et al.  When sound and picture do not fit: Mismatch negativity and sensory interaction. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Jean Vroomen,et al.  Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events , 2007, Journal of Cognitive Neuroscience.

[24]  L. Merabet,et al.  Occipital Transcranial Magnetic Stimulation Has Opposing Effects on Visual and Auditory Stimulus Detection: Implications for Multisensory Interactions , 2007, The Journal of Neuroscience.

[25]  W. Ritter,et al.  The sources of auditory evoked responses recorded from the human scalp. , 1970, Electroencephalography and clinical neurophysiology.

[26]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[27]  Mikko Sams,et al.  A neuromagnetic study of the integration of audiovisual speech in the brain , 1997 .

[28]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[29]  G. J. Thomas.  Experimental study of the influence of vision on sound localization. , 1941 .

[30]  K. Kaga,et al.  Audiovisual phonological mismatch produces early negativity in auditory cortex , 2005, Neuroreport.

[31]  P. Baudonniere,et al.  Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation , 2003, Neuroscience Letters.

[32]  István Winkler,et al.  Processing abstract auditory features in the human auditory cortex , 2003, NeuroImage.

[33]  R. Näätänen,et al.  The mismatch negativity (MMN) in basic research of central auditory processing: A review , 2007, Clinical Neurophysiology.

[34]  Mikko Sams,et al.  Processing of changes in visual speech in the human auditory cortex. , 2002, Brain research. Cognitive brain research.

[35]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[36]  Mikko Sams,et al.  Visual Processing Affects the Neural Basis of Auditory Discrimination , 2008, Journal of Cognitive Neuroscience.

[37]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[38]  P. Deltenre,et al.  Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory , 2002, Clinical Neurophysiology.

[39]  Klucharev Vasily,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003 .

[40]  Jeff Miller,et al.  Storage of feature conjunctions in transient auditory memory. , 1997, Psychophysiology.

[41]  Daniel S. Kislyuk,et al.  The effect of viewing speech on auditory speech processing is different in the left and right hemispheres , 2008, Brain Research.

[42]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[43]  Thomas F Münte,et al.  Temporal dynamics of early perceptual processing , 1998, Current Opinion in Neurobiology.

[44]  T. Takeda,et al.  Modulation of early auditory processing by visually based sound prediction , 2006, Brain Research.

[45]  John J. Foxe,et al.  Multisensory processing and oscillatory activity: analyzing non-linear electrophysiological measures in humans and simians , 2007, Experimental Brain Research.

[46]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[47]  Daniel Senkowski,et al.  Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations , 2007, Neuropsychologia.

[48]  E. Schröger,et al.  Speeded responses to audiovisual signal changes result from bimodal integration. , 1998, Psychophysiology.

[49]  P. Deltenre,et al.  Electrophysiology of spatial scene analysis: the mismatch negativity (MMN) is sensitive to the ventriloquism illusion , 2002, Clinical Neurophysiology.

[50]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[51]  W. Lutzenberger,et al.  Sequential audiovisual interactions during speech perception: A whole-head MEG study , 2007, Neuropsychologia.

[52]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[53]  J. Pernier,et al.  Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. , 2002, Cerebral cortex.

[54]  M. Tervaniemi,et al.  From symbols to sounds: visual symbolic information activates sound representations. , 2004, Psychophysiology.

[55]  G. A. Calvert,et al.  Auditory-visual processing represented in the human superior temporal gyrus , 2007, Neuroscience.

[56]  B. Stein,et al.  The Merging of the Senses , 1993 .

[57]  M. Tervaniemi,et al.  Representation of abstract attributes of auditory stimuli in the human brain. , 1992, Neuroreport.

[58]  J. Kaiser,et al.  Multisensory Object Perception in the Primate Brain , 2010 .

[59]  G Nyman,et al.  Mismatch negativity (MMN) for sequences of auditory and visual stimuli: evidence for a mechanism specific to the auditory modality. , 1990, Electroencephalography and clinical neurophysiology.

[60]  Jean Vroomen,et al.  Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity , 2004, Neuroscience Letters.

[61]  O. Bertrand,et al.  Visual Activation and Audiovisual Interactions in the Auditory Cortex during Speech Perception: Intracranial Recordings in Humans , 2008, The Journal of Neuroscience.

[62]  Ryusuke Kakigi,et al.  Interaction between auditory and visual stimulus relating to the vowel sounds in the auditory cortex in humans: a magnetoencephalographic study , 2004, Neuroscience Letters.

[63]  P. Paavilainen,et al.  Preattentive detection of nonsalient contingencies between auditory features , 2007, Neuroreport.

[64]  M. Sams,et al.  Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans , 2004, Neuroreport.

[65]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[66]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[67]  A. Fort,et al.  Is the auditory sensory memory sensitive to visual information? , 2005, Experimental Brain Research.

[68]  H. Tiitinen,et al.  Visual short-term memory load affects sensory processing of irrelevant sounds in human auditory cortex. , 2003, Brain research. Cognitive brain research.

[69]  Marie Helene Giard,et al.  Methodological Considerations: Electrophysiology of Multisensory Interactions in Humans , 2010 .

[70]  Alexandra Fort,et al.  Interest and validity of the additive model in electrophysiological studies of multisensory interactions , 2004, Cognitive Processing.

[71]  Steven A. Hillyard,et al.  Effects of Spatial Congruity on Audio-Visual Multimodal Integration , 2005, Journal of Cognitive Neuroscience.