Auditory–visual multisensory interactions attenuate subsequent visual responses in humans

Effects of multisensory interactions on how subsequent sensory inputs are processed remain poorly understood. We investigated whether multisensory interactions between rudimentary visual and auditory stimuli (flashes and beeps) affect later visual processing. A 2 x 3 design varied the number of flashes (1 or 2) with the number of beeps (0, 1, or 2) presented on each trial, such that '2F1B' refers to the presentation of 2 flashes with 1 beep. Beeps, when present, were synchronous with the first flash, and pairs of stimuli within a trial were separated by 52 ms ISI. Subjects indicated the number of flashes presented. Electrical neuroimaging of 128-channel event-related potentials assessed both the electric field strength and topography. Isolation of responses a visual stimulus that was preceded by a multisensory event was achieved by calculating the difference between the 2F1B and 1F1B conditions, and responses to a visual stimulus preceded by a unisensory event were isolated by calculating the difference between the 2F0B and 1F0B conditions (MUL and VIS, respectively). Comparison of MUL and VIS revealed that the treatment of visual information was significantly attenuated approximately 160 ms after the onset of the second flash when it was preceded by a multisensory event. Source estimations further indicated that this attenuation occurred within low-level visual cortices. Multisensory interactions are ongoing in low-level visual cortices and affect incoming sensory processing. These data provide evidence that multisensory interactions are not restricted in time and can dramatically influence the treatment of subsequent stimuli, opening new lines of multisensory research.

[1]  A. Giraud,et al.  Implicit Multisensory Associations Influence Voice Recognition , 2006, PLoS biology.

[2]  R Plonsey,et al.  The nature of sources of bioelectric and biomagnetic fields. , 1982, Biophysical journal.

[3]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[4]  D. Jeffreys,et al.  Visual evoked potentials to double-pulse pattern presentation , 1983, Vision Research.

[5]  Mikko Sams,et al.  Perception of matching and conflicting audiovisual speech in dyslexic and fluent readers: An fMRI study at 3 T , 2006, NeuroImage.

[6]  M. Honda,et al.  Behavioral / Systems / Cognitive Functionally Segregated Neural Substrates for Arbitrary Audiovisual Paired-Association Learning , 2005 .

[7]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[8]  Christoph M. Michel,et al.  Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging , 2004, NeuroImage.

[9]  Jean-Philippe Thiran,et al.  Multisensory interactions within human primary cortices revealed by BOLD dynamics. , 2007, Cerebral cortex.

[10]  R. Srebro,et al.  A bootstrap method to compare the shapes of two scalp fields. , 1996, Electroencephalography and clinical neurophysiology.

[11]  Henry Kennedy,et al.  Long-distance feedback projections to area V1: Implications for multisensory integration, spatial awareness, and visual consciousness , 2004, Cognitive, affective & behavioral neuroscience.

[12]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[13]  M. Torrens.  Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268 , 1990 .

[14]  B. Stein,et al.  The Merging of the Senses , 1993 .

[15]  D. Lehmann,et al.  Principles of spatial analysis , 1987 .

[16]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[17]  Joost X. Maier,et al.  Multisensory Integration of Dynamic Faces and Voices in Rhesus Monkey Auditory Cortex , 2005 .

[18]  M. Wallace,et al.  A revised view of sensory cortical parcellation , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[19]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[20]  S. Petersen,et al.  Memory's echo: vivid remembering reactivates sensory-specific cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[21]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[22]  E. Tulving,et al.  Reactivation of encoding-related brain activity during memory retrieval. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Michael D Rugg,et al.  Remembrance of Odors Past Human Olfactory Cortex in Cross-Modal Recognition Memory , 2004, Neuron.

[24]  W. Singer,et al.  Retinotopic effects during spatial audio-visual integration , 2007, Neuropsychologia.

[25]  G. Pfurtscheller.  Handbook of electroencephalography and clinical neurophysiology , 1978 .

[26]  Ladan Shams,et al.  Early modulation of visual cortex by sound: an MEG study , 2005, Neuroscience Letters.

[27]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[28]  D. Guthrie,et al.  Significance testing of difference potentials. , 1991, Psychophysiology.

[29]  R. C. Oldfield.  The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[30]  John J. Foxe,et al.  Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study , 2001, Neuropsychologia.

[31]  John J. Foxe,et al.  Cerebral Cortex Advance Access published March 16, 2005 Filling-in in Schizophrenia: a High-density Electrical Mapping and Source-analysis Investigation of Illusory Contour Processing , 2022 .

[32]  Mikko Sams,et al.  Seeing speech affects acoustic information processing in the human brainstem , 2005, Experimental Brain Research.

[33]  M. Murray,et al.  EEG source imaging , 2004, Clinical Neurophysiology.

[34]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[35]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[36]  J. Gentle,et al.  Randomization and Monte Carlo Methods in Biology. , 1990 .

[37]  G. B. Arden,et al.  Electrical activity in visual cortex associated with combined auditory and visual stimulation in temporal sequences known to be associated with a visual illusion , 2003, Vision Research.

[38]  Dietrich Lehmann,et al.  Mapping event-related brain potential microstates to sentence endings , 2005, Brain Topography.

[39]  Geraint Rees,et al.  Sound alters activity in human V1 in association with illusory visual perception , 2006, NeuroImage.

[40]  Kathleen S Rockland,et al.  Multisensory convergence in calcarine visual areas in macaque monkey. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[41]  J. Maunsell,et al.  Sensory modality specificity of neural activity related to memory in visual cortex. , 1997, Journal of neurophysiology.

[42]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[43]  Christoph M. Michel,et al.  Electrical neuroimaging based on biophysical constraints , 2004, NeuroImage.

[44]  M. Murray,et al.  The role of multisensory memories in unisensory object discrimination. , 2005, Brain research. Cognitive brain research.

[45]  S. Hillyard,et al.  Identification of early visual evoked potential generators by retinotopic and topographic analyses , 1994 .

[46]  M. Wallace,et al.  Early experience determines how the senses will interact. , 2007, Journal of neurophysiology.

[47]  Micah M. Murray,et al.  The brain uses single-trial multisensory memories to discriminate without awareness , 2005, NeuroImage.

[48]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[49]  John J. Foxe,et al.  Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study. , 2002, Journal of neurophysiology.

[50]  G. V. Simpson,et al.  Flow of activation from V1 to frontal cortex in humans , 2001, Experimental Brain Research.

[51]  C. Schroeder,et al.  Somatosensory input to auditory association cortex in the macaque monkey. , 2001, Journal of neurophysiology.

[52]  Paul J. Laurienti,et al.  Semantic congruence is a critical factor in multisensory behavioral performance , 2004, Experimental Brain Research.

[53]  C. Michel,et al.  Noninvasive Localization of Electromagnetic Epileptic Activity. I. Method Descriptions and Simulations , 2004, Brain Topography.

[54]  John J. Foxe,et al.  Boundary Completion Is Automatic and Dissociable from Shape Discrimination , 2006, The Journal of Neuroscience.

[55]  F. Perrin,et al.  Mapping of scalp potentials by surface spline interpolation. , 1987, Electroencephalography and clinical neurophysiology.

[56]  D. Lehmann,et al.  Segmentation of brain electrical activity into microstates: model estimation and validation , 1995, IEEE Transactions on Biomedical Engineering.

[57]  John J. Foxe,et al.  Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. , 2005, Cerebral cortex.

[58]  P. Barone,et al.  Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey , 2005, The European journal of neuroscience.

[59]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[60]  S. Hillyard,et al.  Involvement of striate and extrastriate visual cortical areas in spatial attention , 1999, Nature Neuroscience.

[61]  M. Sams,et al.  Primary auditory cortex activation by visual speech: an fMRI study at 3 T , 2005, Neuroreport.

[62]  N. Logothetis,et al.  Integration of Touch and Sound in Auditory Cortex , 2005, Neuron.

[63]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[64]  W. Skrandies,et al.  Cortical and retinal refractory periods in the human visual system. , 1989, The International journal of neuroscience.

[65]  D. Lehmann,et al.  Reference-free identification of components of checkerboard-evoked multichannel potential fields. , 1980, Electroencephalography and clinical neurophysiology.

[66]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[67]  Micah M. Murray,et al.  Rapid Brain Discrimination of Sounds of Objects , 2006, The Journal of Neuroscience.

[68]  Mikko Sams,et al.  Factors influencing audiovisual fission and fusion illusions. , 2004, Brain research. Cognitive brain research.