Matching Auditory and Visual Signals: Is Sensory Modality Just Another Feature?

In order to perceive the world coherently, we need to integrate features of objects and events that are presented to our senses. Here we investigated the temporal limit of integration in unimodal visual and auditory as well as crossmodal auditory–visual conditions. Participants were presented with alternating visual and auditory stimuli and were asked to match them either within or between modalities. At alternation rates of about 4 Hz and higher, participants were no longer able to match visual and auditory stimuli across modalities correctly, while matching within either modality showed higher temporal limits. Manipulating different temporal stimulus characteristics (stimulus offsets and/or auditory–visual SOAs) did not change performance. Interestingly, the difference in temporal limits between crossmodal and unimodal conditions appears strikingly similar to temporal limit differences between unimodal conditions when additional features have to be integrated. We suggest that adding a modality across which sensory input is integrated has the same effect as adding an extra feature to be integrated within a single modality.

[1]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[2]  G H MOWBRAY,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959, The American journal of psychology.

[3]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[4]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[5]  Marisa Carrasco,et al.  Temporal performance fields: visual and attentional factors , 2004, Vision Research.

[6]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[7]  David M. Green,et al.  Temporal Factors in Psychoacoustics , 1985 .

[8]  Simon Carlile,et al.  Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[9]  David Alais,et al.  Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. , 2006, Journal of vision.

[10]  Patrick Cavanagh,et al.  Early binding of feature pairs for visual perception , 2001, Nature Neuroscience.

[11]  R Fendrich,et al.  The temporal cross-capture of audition and vision , 2001, Perception & psychophysics.

[12]  Y. Yeshurun,et al.  Transient Spatial Attention Degrades Temporal Resolution , 2003, Psychological science.

[13]  Shin'ya Nishida,et al.  Visual search for a target changing in synchrony with an auditory signal , 2006, Proceedings of the Royal Society B: Biological Sciences.

[14]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[15]  Mowbray Gh,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959 .

[16]  Derek H. Arnold,et al.  Timing sight and sound , 2005, Vision Research.

[17]  J. Reynolds,et al.  Temporal Resolution for the Perception of Features and Conjunctions , 2007, The Journal of Neuroscience.

[18]  Waka Fujisaki,et al.  Temporal frequency characteristics of synchrony–asynchrony discrimination of audio-visual signals , 2005, Experimental Brain Research.

[19]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[20]  C. Spence,et al.  Audiovisual temporal order judgments , 2003, Experimental Brain Research.

[21]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[22]  E. de Boer,et al.  Auditory Time Constants: A Paradox? , 1985 .

[23]  S. Zeki,et al.  The temporal order of binding visual attributes , 2006, Vision Research.

[24]  G. Recanzone.  Auditory influences on visual temporal rate perception. , 2003, Journal of neurophysiology.

[25]  P. Stopka.  Implicit Estimation of Sound-arrival Time , 2022 .

[26]  C. Spence,et al.  Visual Prior Entry , 2001, Psychological science.

[27]  C. Spence,et al.  Audiovisual prior entry , 2003, Neuroscience Letters.

[28]  R. Fendrich,et al.  Closer in time when farther in space--spatial factors in audiovisual temporal integration. , 2005, Brain research. Cognitive brain research.

[29]  Jean Vroomen,et al.  The role of spatial disparity and hemifields in audio-visual temporal order judgments , 2005, Experimental Brain Research.

[30]  N. Viemeister,et al.  Temporal integration and multiple looks. , 1991, The Journal of the Acoustical Society of America.