Effect of Attention on Early Cortical Processes Associated with the Sound-induced Extra Flash Illusion

When a single flash of light is presented interposed between two brief auditory stimuli separated by 60–100 msec, subjects typically report perceiving two flashes [Shams, L., Kamitani, Y., & Shimojo, S. Visual illusion induced by sound. Brain Research, Cognitive Brain Research, 14, 147–152, 2002; Shams, L., Kamitani, Y., & Shimojo, S. Illusions. What you see is what you hear. Nature, 408, 788, 2000]. Using ERP recordings, we previously found that perception of the illusory extra flash was accompanied by a rapid dynamic interplay between auditory and visual cortical areas that was triggered by the second sound [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. In the current study, we investigated the effect of attention on the ERP components associated with the illusory extra flash in 15 individuals who perceived this cross-modal illusion frequently. All early ERP components in the cross-modal difference wave associated with the extra flash illusion were significantly enhanced by selective spatial attention. The earliest attention-related modulation was an amplitude increase of the positive-going PD110/PD120 component, which was previously shown to be correlated with an individual's propensity to perceive the illusory second flash [Mishra, J., Martínez, A., Sejnowski, T. J., & Hillyard, S. A. Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27, 4120–4131, 2007]. The polarity of the early PD110/PD120 component did not differ as a function of the visual field (upper vs. lower) of stimulus presentation. This, along with the source localization of the component, suggested that its principal generator lies in extrastriate visual cortex. These results indicate that neural processes previously shown to be associated with the extra flash illusion can be modulated by attention, and thus are not the result of a wholly automatic cross-modal integration process.

[1]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[2]  Steve Steiner,et al.  Missing link , 1989 .

[3]  S A Hillyard,et al.  Electrophysiological evidence for the "missing link" in crossmodal attention. , 2001, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[4]  G. Mangun,et al.  The neural mechanisms of top-down attentional control , 2000, Nature Neuroscience.

[5]  S. Hillyard,et al.  Neural Basis of the Ventriloquist Illusion , 2007, Current Biology.

[6]  M. Carrasco.  Covert attention increases contrast sensitivity: Psychophysical, neurophysiological and neuroimaging studies. , 2006, Progress in brain research.

[7]  Antigona Martínez,et al.  Source analysis of event-related cortical activity during visuo-spatial attention. , 2003, Cerebral cortex.

[8]  R Kakigi,et al.  [Event-related brain potentials]. , 1997, Nihon rinsho. Japanese journal of clinical medicine.

[9]  I..  THE ATTENTION SYSTEM OF THE HUMAN BRAIN , 2002 .

[10]  S. Hillyard,et al.  Event-related brain potentials in the study of visual selective attention. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[12]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[13]  Sabine Kastner,et al.  Visual attention as a multilevel selection process , 2004, Cognitive, affective & behavioral neuroscience.

[14]  T. Sejnowski,et al.  Early Cross-Modal Interactions in Auditory and Visual Cortex Underlie a Sound-Induced Visual Illusion , 2007, The Journal of Neuroscience.

[15]  Viola S. Störmer,et al.  Cross-Modal Spatial Cueing of Attention Influences Visual Perception , 2012 .

[16]  S. Hillyard,et al.  Identification of early visual evoked potential generators by retinotopic and topographic analyses , 1994 .

[17]  Durk Talsma,et al.  Intermodal spatial attention differs between vision and audition: An event-related potential analysis , 2002 .

[18]  R Fendrich,et al.  The temporal cross-capture of audition and vision , 2001, Perception & psychophysics.

[19]  S. Hillyard,et al.  Neural basis of auditory-induced shifts in visual time-order perception , 2005, Nature Neuroscience.

[20]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[21]  P. Bertelson.  Chapter 14 Ventriloquism: A case of crossmodal perceptual grouping , 1999 .

[22]  Jeffreys Da,et al.  Separable components of human evoked responses to spatially patterned visual fields. , 1968 .

[23]  Shinsuke Shimojo,et al.  Sound-induced illusory flash perception: role of gamma band responses , 2002, Neuroreport.

[24]  Leslie G. Ungerleider,et al.  Mechanisms of visual attention in the human cortex. , 2000, Annual review of neuroscience.

[25]  J. Reynolds,et al.  Attentional modulation of visual processing. , 2004, Annual review of neuroscience.

[26]  C. Rorden,et al.  Stereotaxic display of brain lesions. , 2000, Behavioural neurology.

[27]  S. Hillyard,et al.  Intra-modal and cross-modal spatial attention to auditory and visual stimuli. An event-related brain potential study. , 1999, Brain research. Cognitive brain research.

[28]  Robert Oostenveld,et al.  Perception of the touch-induced visual double-flash illusion correlates with changes of rhythmic neuronal activity in human visual and somatosensory areas , 2011, NeuroImage.

[29]  Jon Driver,et al.  ERP Evidence for Cross-Modal Audiovisual Effects of Endogenous Spatial Attention within Hemifields , 2004, Journal of Cognitive Neuroscience.

[30]  J. Driver,et al.  Crossmodal links in endogenous and exogenous spatial attention: evidence from event-related brain potential studies , 2001, Neuroscience & Biobehavioral Reviews.

[31]  John J. Foxe,et al.  Object‐based attention is multisensory: co‐activation of an object's representations in ignored sensory modalities , 2007, The European journal of neuroscience.

[32]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[33]  S. Hillyard,et al.  Cortical sources of the early components of the visual evoked potential , 2002, Human brain mapping.

[34]  Martin Eimer,et al.  Can attention be directed to opposite locations in different modalities? An ERP study , 1999, Clinical Neurophysiology.

[35]  Steven A. Hillyard,et al.  Effects of Spatial Congruity on Audio-Visual Multimodal Integration , 2005, Journal of Cognitive Neuroscience.

[36]  Steven A. Hillyard,et al.  Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. , 1994 .

[37]  Rolf Ulrich,et al.  Illusory double flashes can speed up responses like physical ones: evidence from the sound-induced flash illusion , 2011, Experimental Brain Research.

[38]  Richard B Buxton,et al.  Putting spatial attention on the map: timing and localization of stimulus selection processes in striate and extrastriate visual areas , 2001, Vision Research.

[39]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[40]  E. Macaluso,et al.  Multisensory spatial interactions: a window onto functional integration in the human brain , 2005, Trends in Neurosciences.

[41]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[42]  Brigitte Röder,et al.  A new method for detecting interactions between the senses in event-related potentials , 2006, Brain Research.

[43]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[44]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[45]  H. Kennedy,et al.  Anatomical Evidence of Multimodal Integration in Primate Striate Cortex , 2002, The Journal of Neuroscience.

[46]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[47]  Geraint Rees,et al.  Activity in human V1 follows multisensory perception , 2007, NeuroImage.

[48]  E. Bizzi,et al.  The Cognitive Neurosciences , 1996 .

[49]  Antígona Martínez,et al.  Spatial attention facilitates selection of illusory objects: Evidence from event-related brain potentials , 2007, Brain Research.

[50]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[51]  G. B. Arden,et al.  Electrical activity in visual cortex associated with combined auditory and visual stimulation in temporal sequences known to be associated with a visual illusion , 2003, Vision Research.

[52]  Ladan Shams,et al.  Early modulation of visual cortex by sound: an MEG study , 2005, Neuroscience Letters.

[53]  Geraint Rees,et al.  Sound alters activity in human V1 in association with illusory visual perception , 2006, NeuroImage.

[54]  Kathleen S Rockland,et al.  Multisensory convergence in calcarine visual areas in macaque monkey. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  E. Vogel,et al.  Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[56]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[57]  Steven A. Hillyard,et al.  Objects Are Highlighted by Spatial Attention , 2006 .

[58]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[59]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[60]  E. Schröger,et al.  ERP effects of intermodal attention and cross-modal links in spatial attention. , 1998, Psychophysiology.

[61]  A. Amedi,et al.  Functional imaging of human crossmodal identification and object recognition , 2005, Experimental Brain Research.

[62]  Steven A. Hillyard,et al.  Neural Substrates of Perceptual Enhancement by Cross-Modal Spatial Attention , 2003, Journal of Cognitive Neuroscience.

[63]  Durk Talsma,et al.  Attentional orienting across the sensory modalities , 2008, Brain and Cognition.

[64]  G. Recanzone.  Auditory influences on visual temporal rate perception. , 2003, Journal of neurophysiology.

[65]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[66]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[67]  S. Hillyard,et al.  Human auditory evoked potentials. I. Evaluation of components. , 1974, Electroencephalography and clinical neurophysiology.

[68]  Jean Vroomen,et al.  Perceptual effects of cross-modal stimulation : The cases of ventriloquism and the freezing phenomenon , 2004 .

[69]  R. Burkard.  Human Auditory Evoked Potentials , 2010 .

[70]  B. Stein,et al.  The Merging of the Senses , 1993 .

[71]  Henry Kennedy,et al.  Long-distance feedback projections to area V1: Implications for multisensory integration, spatial awareness, and visual consciousness , 2004, Cognitive, affective & behavioral neuroscience.

[72]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[73]  M. Wallace,et al.  Visual Localization Ability Influences Cross-Modal Bias , 2003, Journal of Cognitive Neuroscience.

[74]  C. C. Wood,et al.  Scalp distributions of event-related potentials: an ambiguity associated with analysis of variance models. , 1985, Electroencephalography and clinical neurophysiology.

[75]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[76]  L. Busse,et al.  The spread of attention across modalities and space in a multisensory object. , 2005, Proceedings of the National Academy of Sciences of the United States of America.