Effects of Auditory Stimuli in the Horizontal Plane on Audiovisual Integration: An Event-Related Potential Study

This article aims to investigate whether auditory stimuli in the horizontal plane, particularly originating from behind the participant, affect audiovisual integration by using behavioral and event-related potential (ERP) measurements. In this study, visual stimuli were presented directly in front of the participants, auditory stimuli were presented at one location in an equidistant horizontal plane at the front (0°, the fixation point), right (90°), back (180°), or left (270°) of the participants, and audiovisual stimuli that include both visual stimuli and auditory stimuli originating from one of the four locations were simultaneously presented. These stimuli were presented randomly with equal probability; during this time, participants were asked to attend to the visual stimulus and respond promptly only to visual target stimuli (a unimodal visual target stimulus and the visual target of the audiovisual stimulus). A significant facilitation of reaction times and hit rates was obtained following audiovisual stimulation, irrespective of whether the auditory stimuli were presented in the front or back of the participant. However, no significant interactions were found between visual stimuli and auditory stimuli from the right or left. Two main ERP components related to audiovisual integration were found: first, auditory stimuli from the front location produced an ERP reaction over the right temporal area and right occipital area at approximately 160–200 milliseconds; second, auditory stimuli from the back produced a reaction over the parietal and occipital areas at approximately 360–400 milliseconds. Our results confirmed that audiovisual integration was also elicited, even though auditory stimuli were presented behind the participant, but no integration occurred when auditory stimuli were presented in the right or left spaces, suggesting that the human brain might be particularly sensitive to information received from behind than both sides.

[1]  Jeffery A. Jones,et al.  Multisensory integration of speech signals: the relationship between space and time , 2006, Experimental Brain Research.

[2]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[3]  Michael D. Rugg,et al.  Word and Nonword Repetition Within- and Across-Modality: An Event-Related Potential Study , 1995, Journal of Cognitive Neuroscience.

[4]  Mikko Sams,et al.  Audio–visual speech perception is special , 2005, Cognition.

[5]  M. Frens,et al.  Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements , 1995, Perception & psychophysics.

[6]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[7]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[8]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[9]  Lynne E. Bernstein,et al.  Spatiotemporal dynamics of audiovisual speech processing , 2008, NeuroImage.

[10]  Carlo Umiltà,et al.  Crossmodal object-based attention: Auditory objects affect visual processing , 2005, Cognition.

[11]  Micah M. Murray,et al.  Auditory–somatosensory multisensory interactions in front and rear space , 2007, Neuropsychologia.

[12]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[13]  A D Musicant,et al.  The influence of pinnae-based spectral cues on sound localization. , 1984, The Journal of the Acoustical Society of America.

[14]  H Stanislaw,et al.  Calculation of signal detection theory measures , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[15]  S A Hillyard,et al.  An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. , 2002, Brain research. Cognitive brain research.

[16]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[17]  Hiroshi Ando,et al.  Neural correlates of sound externalization , 2013, NeuroImage.

[18]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[19]  Rick L. Jenison,et al.  Auditory Spatial Layout , 1995 .

[20]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[21]  Alexandra Fort,et al.  Interest and validity of the additive model in electrophysiological studies of multisensory interactions , 2004, Cognitive Processing.

[22]  Steven A. Hillyard,et al.  Effects of Spatial Congruity on Audio-Visual Multimodal Integration , 2005, Journal of Cognitive Neuroscience.

[23]  J. Pernier,et al.  Early auditory-visual interactions in human cortex during nonredundant target identification. , 2002, Brain research. Cognitive brain research.

[24]  Tetsuo Touge,et al.  Audiovisual interaction enhances auditory detection in late stage: an event-related potential study , 2010, Neuroreport.

[25]  C. K. Peck,et al.  Spatial disparity affects visual-auditory interactions in human sensorimotor processing , 1998, Experimental Brain Research.

[26]  M. Morimoto,et al.  The contribution of two ears to the perception of vertical angle in sagittal planes. , 2001, The Journal of the Acoustical Society of America.

[27]  K. Pelphrey,et al.  Cortical integration of audio–visual speech and non-speech stimuli , 2010, Brain and Cognition.

[28]  Michael T. Lippert,et al.  Improvement of visual contrast detection by a simultaneous sound , 2007, Brain Research.

[29]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[30]  Zachary P. Barnett,et al.  Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance , 2012, Experimental Brain Research.

[31]  David Poeppel,et al.  Visual speech speeds up the neural processing of auditory speech. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[32]  S. Hillyard,et al.  Involuntary orienting to sound improves visual perception , 2000, Nature.

[33]  Brigitte Röder,et al.  Multisensory processing in the redundant-target effect: A behavioral and event-related potential study , 2005, Perception & psychophysics.

[34]  J. Fell,et al.  Lateralized auditory spatial perception and the contralaterality of cortical processing as studied with functional magnetic resonance imaging and magnetoencephalography , 1999, Human brain mapping.

[35]  Kestutis Kveraga,et al.  Multimodal access to verbal name codes , 2007, Perception & psychophysics.

[36]  Tetsuo Touge,et al.  Multisensory Interactions Elicited by Audiovisual Stimuli Presented Peripherally in a Visual Attention Task: A Behavioral and Event-Related Potential Study in Humans , 2009, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[37]  A. Fort,et al.  Bimodal speech: early suppressive visual effects in human auditory cortex , 2004, The European journal of neuroscience.

[38]  D. Barth,et al.  The spatiotemporal organization of auditory, visual, and auditory-visual evoked potentials in rat cortex , 1995, Brain Research.

[39]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[40]  David B. Pisoni,et al.  Neural processing of asynchronous audiovisual speech perception , 2010, NeuroImage.

[41]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[42]  S Baron-Cohen,et al.  The physiology of coloured hearing. A PET activation study of colour-word synaesthesia. , 1995, Brain : a journal of neurology.

[43]  N. Bolognini,et al.  Enhancement of visual perception by crossmodal visuo-auditory interaction , 2002, Experimental Brain Research.

[44]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[45]  J. Theeuwes,et al.  Attention and the multiple stages of multisensory integration: A review of audiovisual studies. , 2010, Acta psychologica.

[46]  Hans Colonius,et al.  Time-Window-of-Integration (TWIN) Model for Saccadic Reaction Time: Effect of Auditory Masker Level on Visual–Auditory Spatial Interaction in Elevation , 2009, Brain Topography.

[47]  J. Pernier,et al.  Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans. , 2002, Cerebral cortex.