The effect of temporal asynchrony on audiovisual integration

Many studies have shown that temporal asynchrony of visual-auditory stimuli can influence the integration between visual and auditory modality, however, the multisensory mechanisms of asynchronous inputs were not well understood. To investigate the effect of temporal asynchrony on audiovisual integration, we measured event-related potentials (ERPs) and behavior in human subjects. In this study, visual and auditory stimuli onset asynchronous (SOA= ±100 ms, ±50 ms, 0 ms), both the auditory and visual stimuli were attended. Using event-related potential, the differences of audiovisual integration elicited by asynchronous audiovisual stimuli were found at 140-200 ms over a wide parietal and occipital regions. These results revealed the asynchronous audiovisual stimuli enhanced the detection and elicited audiovisual integration. We expect that these results could helpful to process the audiovisual information of artificial intelligence.

[1]  G. Recanzone,et al.  Temporal and spatial dependency of the ventriloquism effect , 2001, Neuroreport.

[2]  Donald A Hodges,et al.  Auditory enhancement of visual temporal order judgment , 2006, Neuroreport.

[3]  Tetsuo Touge,et al.  Audiovisual interaction enhances auditory detection in late stage: an event-related potential study , 2010, Neuroreport.

[4]  Tetsuo Touge,et al.  Multisensory Interactions Elicited by Audiovisual Stimuli Presented Peripherally in a Visual Attention Task: A Behavioral and Event-Related Potential Study in Humans , 2009, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[5]  Christopher T. Lovelace,et al.  An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. , 2003, Brain research. Cognitive brain research.

[6]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[7]  L M Ward,et al.  Involuntary Listening Aids Seeing: Evidence From Human Electrophysiology , 2000, Psychological science.

[8]  M. Wallace,et al.  Multisensory integration in the superior colliculus of the alert cat. , 1998, Journal of neurophysiology.

[9]  M. Carrasco,et al.  Attention alters appearance , 2004, Nature Neuroscience.

[10]  M. Frens,et al.  Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements , 1995, Perception & psychophysics.

[11]  J. Theeuwes,et al.  Attention and the multiple stages of multisensory integration: A review of audiovisual studies. , 2010, Acta psychologica.

[12]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[13]  Beijing Beijing,et al.  IEEE International Conference on Mechatronics and Automation , 2015 .

[14]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[15]  Michael T. Lippert,et al.  Improvement of visual contrast detection by a simultaneous sound , 2007, Brain Research.

[16]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[17]  E. Vogel,et al.  The visual N1 component as an index of a discrimination process. , 2000, Psychophysiology.

[18]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[19]  M. Wallace,et al.  Visual Localization Ability Influences Cross-Modal Bias , 2003, Journal of Cognitive Neuroscience.

[20]  M. Meredith,et al.  On the neuronal basis for multisensory convergence: a brief overview. , 2002, Brain research. Cognitive brain research.

[21]  N. Bolognini,et al.  Enhancement of visual perception by crossmodal visuo-auditory interaction , 2002, Experimental Brain Research.

[22]  N. Bruneau,et al.  Cross-modal processing of auditory–visual stimuli in a no-task paradigm: A topographic event-related potential study , 2008, Clinical Neurophysiology.

[23]  Hans Colonius,et al.  Visual-tactile spatial interaction in saccade generation , 2003, Experimental Brain Research.