Audio-visual simultaneity judgments

The relative spatiotemporal correspondence between sensory events affects multisensory integration across a variety of species; integration is maximal when stimuli in different sensory modalities are presented from approximately the same position at about the same time. In the present study, we investigated the influence of spatial and temporal factors on audio-visual simultaneity perception in humans. Participants made unspeeded simultaneous versus successive discrimination responses to pairs of auditory and visual stimuli presented at varying stimulus onset asynchronies from either the same or different spatial positions using either the method of constant stimuli (Experiments 1 and 2) or psychophysical staircases (Experiment 3). The participants in all three experiments were more likely to report the stimuli as being simultaneous when they originated from the same spatial position than when they came from different positions, demonstrating that the apparent perception of multisensory simultaneity is dependent on the relative spatial position from which stimuli are presented.

[1]  E. C. Sanford.  Minor Studies from the Psychological Laboratory of Clark University , 1893 .

[2]  G. Whipple.  On Nearly Simultaneous Clicks and Flashes , 1899 .

[3]  M. H. Pirenne,et al.  The sensations : their functions, processes and mechanisms , 1952 .

[4]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[5]  P. Fraisse.  The psychology of time , 1963 .

[6]  I. Hirsh,et al.  [SIMULTANEOUS CHARACTER AND SUCCESSION OF HETEROGENOUS STIMULI]. , 1964, L'annee psychologique.

[7]  P. Fraisse,et al.  Simultanéité et succession de stimuli hétérogènes , 1964 .

[8]  A B Kristofferson,et al.  Attention and psychophysical time. , 1967, Acta psychologica.

[9]  The frequency principle for comparative judgments) , 1967 .

[10]  M. M. Taylor,et al.  PEST: Efficient Estimates on Probability Functions , 1967 .

[11]  R. Sekuler,et al.  The invalidity of “invalid results from the method of constant stimuli”: A common artifact in the methods of psychophysics , 1971 .

[12]  R. Sekuler,et al.  Response frequency equalization: A bias model for psychophysics , 1971 .

[13]  G. R. Engel,et al.  Visual-Auditory Distance Constancy , 1971, Nature.

[14]  E. A. Miller.  Interaction of vision and touch in conflict and nonconflict form perception tasks. , 1972, Journal of experimental psychology.

[15]  W R Thurlow,et al.  Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. , 1973, Perceptual and motor skills.

[16]  L. Allan.  The relationship between judgments ofsuccessiveness and judgments of order , 1975 .

[17]  Robert B. Welch,et al.  The “ventriloquist effect”: Visual dominance or response bias? , 1975 .

[18]  L. Allan.  The relationship between judgments of successiveness , 1975 .

[19]  P. Bertelson,et al.  Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations , 1977 .

[20]  G. Shulman,et al.  Moving attention through visual space. , 1979, Journal of experimental psychology. Human perception and performance.

[21]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[22]  R. Campbell,et al.  Hearing by Eye , 1980, The Quarterly journal of experimental psychology.

[23]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[24]  P. A. Kolers,et al.  Sensory specificity of apparent motion. , 1981, Journal of experimental psychology. Human perception and performance.

[25]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[26]  H Kunov,et al.  Disruptive effects of auditory signal delay on speech perception with lipreading. , 1986, The Journal of auditory research.

[27]  P Bertelson,et al.  Auditory-visual interaction and the timing of inputs , 1987, Psychological research.

[28]  G. Rhodes.  Auditory attention and the representation of spatial information , 1987, Perception & psychophysics.

[29]  Piotr Jaśkowski,et al.  Temporal-order judgments and reaction time for stimuli of different modalities , 1990, Psychological research.

[30]  T Z Strybel,et al.  A comparison of the effects of spatial separation on apparent motion in the auditory and visual modalities , 1990, Perception & psychophysics.

[31]  B. Stein,et al.  The Merging of the Senses , 1993 .

[32]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[33]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[34]  J. D. Mollon,et al.  Errors of judgement at Greenwich in 1796 , 1996, Nature.

[35]  Béatrice de Gelder,et al.  Auditory-visual interaction in voice localization and in bimodal speech recognition: the effects of desynchronization , 1997, AVSP.

[36]  Kevin G. Munhall,et al.  Effects of separating auditory and visual sources on audiovisual integration of speech , 1997 .

[37]  G. Aschersleben,et al.  Automatic visual bias of perceived auditory location , 1998 .

[38]  Michael J Brammer,et al.  Crossmodal identification , 1998, Trends in Cognitive Sciences.

[39]  R. Welch.  Chapter 15 Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions , 1999 .

[40]  R. Welch.  The advantages and limitations of the psychophysical staircases procedure in the study of intersensory bias: Commentary on Bertelson , 1999 .

[41]  T. Bachmann,et al.  Cognitive contributions to the perception of spatial and temporal events (Advances in Psychology, Vol. 129) , 1999 .

[42]  P. Jaśkowski.  Chapter 11 Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations , 1999 .

[43]  P. Bertelson.  Chapter 14 Ventriloquism: A case of crossmodal perceptual grouping , 1999 .

[44]  J. Vroomen.  Ventriloquism and the nature of the unity decision: Commentary on Welch , 1999 .

[45]  J. Müsseler,et al.  Ventriloquism and the nature of the unity assumption , 1999 .

[46]  C. Spence,et al.  Multisensory perception: Beyond modularity and convergence , 2000, Current Biology.

[47]  J L Schwartz,et al.  Audio-visual enhancement of speech in noise. , 2001, The Journal of the Acoustical Society of America.

[48]  C. Spence,et al.  Multisensory prior entry. , 2001, Journal of experimental psychology. General.

[49]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[50]  S. Charles,et al.  10 - Crossmodal Attentional Capture: A Controversy Resolved? , 2001 .

[51]  M. Hallett,et al.  Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection , 2001, The Journal of Neuroscience.

[52]  James V. Stone,et al.  When is now? Perception of simultaneity , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[53]  G. Recanzone,et al.  Temporal and spatial dependency of the ventriloquism effect , 2001, Neuroreport.

[54]  Jörg Lewald,et al.  Spatio-temporal constraints for auditory–visual integration , 2001, Behavioural Brain Research.

[55]  Stephen M. Rao,et al.  The evolution of brain activation during temporal processing , 2001, Nature Neuroscience.

[56]  C. Spence,et al.  Confusing the mind by crossing the hands. , 2002, Brain research. Cognitive brain research.

[57]  C. Spence,et al.  Tactile “capture” of audition , 2002, Perception & psychophysics.

[58]  S. Kosslyn,et al.  Impact of fMRI Acoustic Noise on the Functional Anatomy of Visual Mental Imagery , 2002, Journal of Cognitive Neuroscience.

[59]  Yôiti Suzuki,et al.  Implicit estimation of sound-arrival time , 2003, Nature.

[60]  N. Troje,et al.  Audiovisual phenomenal causality , 2003, Perception & psychophysics.

[61]  R. Baddeley,et al.  Multisensory temporal order judgments: When two locations are better than one , 2003, Perception & psychophysics.

[62]  C. Spence,et al.  Multisensory Integration: Maintaining the Perception of Synchrony , 2003, Current Biology.

[63]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[64]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[65]  G. Aschersleben,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[66]  Charles Spence,et al.  Multisensory temporal order judgments: the role of hemispheric redundancy. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[67]  C. Spence,et al.  Audiovisual temporal order judgments , 2003, Experimental Brain Research.

[68]  P. Bertelson,et al.  The psychology of multimodal perception , 2004 .

[69]  E Macaluso,et al.  Spatial and temporal factors during processing of audiovisual speech: a PET study , 2004, NeuroImage.

[70]  C. Spence,et al.  Crossmodal Space and Crossmodal Attention , 2004 .

[71]  Charles Spence,et al.  Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. , 2005, Brain research. Cognitive brain research.

[72]  Sigmund Exner,et al.  Experimentelle Untersuchung der einfachsten psychischen Processe , 1875, Archiv für die gesamte Physiologie des Menschen und der Tiere.