Early modulation of visual cortex by sound: an MEG study

Sound can alter visual perception. This has been recently demonstrated by a strong illusion in which a single flash is perceived as multiple flashes when accompanied by multiple brief sounds. While psychophysical findings on this sound-induced flash illusion indicate that the modulations of visual percept by sound occur at a perceptual processing level, it remains unclear at what level of perceptual processing these interactions occur and what mechanisms mediate them. Here we investigated these questions using MEG. We found modulation of activity in occipital and parietal scalp locations, when comparing illusion trials with visual-alone and auditory-alone trials. This modulation occurred as early as 35-65 ms from the onset of the visual stimulus. Activity was also modulated in the occipital and parietal areas as well as anterior areas at a later ( approximately 150 ms post-stimulus) onset. No significant interactions were observed in occipital and parietal areas in trials in which illusion was not perceived. These results indicate that the auditory alteration of visual perception as reflected by the illusion is associated with modulation of activity in visual cortex. The early onset of these modulations suggests that a feed-forward or lateral circuitry is at least partially involved in these interactions.

[1]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[2]  G. B. Arden,et al.  Electrical activity in visual cortex associated with combined auditory and visual stimulation in temporal sequences known to be associated with a visual illusion , 2003, Vision Research.

[3]  Robert B. Welch,et al.  Contributions of audition and vision to temporal rate perception , 1986, Perception & psychophysics.

[4]  Scott T. Grafton,et al.  Feeling with the mind's eye , 1997, Neuroreport.

[5]  John J. Foxe,et al.  The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. , 2002, Brain research. Cognitive brain research.

[6]  Kathleen S Rockland,et al.  Multisensory convergence in calcarine visual areas in macaque monkey. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[7]  T. Hendler,et al.  Visuo-haptic object-related activation in the ventral visual pathway , 2001, Nature Neuroscience.

[8]  E. Schröger,et al.  Speeded responses to audiovisual signal changes result from bimodal integration. , 1998, Psychophysiology.

[9]  David J. Heeger,et al.  Neuronal correlates of perception in early visual cortex , 2003, Nature Neuroscience.

[10]  J. T. Walker,et al.  Auditory-visual conflicts in the perceived duration of lights, tones and gaps. , 1981, Journal of experimental psychology. Human perception and performance.

[11]  O W Sakowitz,et al.  Bisensory stimulation increases gamma-responses over multiple cortical regions. , 2001, Brain research. Cognitive brain research.

[12]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[13]  Jon Driver,et al.  Crossmodal attention , 1998, Current Opinion in Neurobiology.

[14]  S. Hillyard,et al.  Involuntary orienting to sound improves visual perception , 2000, Nature.

[15]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[16]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[17]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[18]  Scott T. Grafton,et al.  Involvement of visual cortex in tactile discrimination of orientation , 1999, Nature.

[19]  G H MOWBRAY,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959, The American journal of psychology.

[20]  L M Ward,et al.  Involuntary Listening Aids Seeing: Evidence From Human Electrophysiology , 2000, Psychological science.

[21]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[22]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[23]  Ankoor S. Shah,et al.  Auditory Cortical Neurons Respond to Somatosensory Stimulation , 2003, The Journal of Neuroscience.

[24]  Mowbray Gh,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959 .

[25]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[26]  Alan Kingstone,et al.  The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. , 2002, Brain research. Cognitive brain research.

[27]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[28]  J. Driver.  Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading , 1996, Nature.

[29]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[30]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[31]  S. Wuerger,et al.  Cross-modal integration of auditory and visual motion signals , 2001, Neuroreport.

[32]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[33]  C. Lammertmann,et al.  Magnetoencephalographic Correlates of Audiotactile Interaction , 2002, NeuroImage.

[34]  J. Pernier,et al.  Early auditory-visual interactions in human cortex during nonredundant target identification. , 2002, Brain research. Cognitive brain research.

[35]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[36]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[37]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[38]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[39]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[40]  G. Calvert,et al.  Multisensory integration: methodological approaches and emerging principles in the human brain , 2004, Journal of Physiology-Paris.

[41]  Shinsuke Shimojo,et al.  Sound-induced illusory flash perception: role of gamma band responses , 2002, Neuroreport.

[42]  P. Baudonniere,et al.  Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation , 2003, Neuroscience Letters.

[43]  Stephen A. Engel,et al.  Erratum: An oblique effect in human primary visual cortex (nature neuroscience (2000) 3 (535-536)) , 2000 .

[44]  S. Mateeff,et al.  Dynamic Visual Capture: Apparent Auditory Motion Induced by a Moving Visual Target , 1985, Perception.

[45]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[46]  S. Engel,et al.  Interocular rivalry revealed in the human cortical blind-spot representation , 2001, Nature.