Perception of intersensory synchrony: A tutorial review

For most multisensory events, observers perceive synchrony among the various senses (vision, audition, touch), despite the naturally occurring lags in arrival and processing times of the different information streams. A substantial amount of research has examined how the brain accomplishes this. In the present article, we review several key issues about intersensory timing, and we identify four mechanisms of how intersensory lags might be dealt with: by ignoring lags up to some point (a wide window of temporal integration), by compensating for predictable variability, by adjusting the point of perceived synchrony on the longer term, and by shifting one stream directly toward the other.

[1]  W. F. Smith The relative quickness of visual and auditory perception. , 1933 .

[2]  F. K. Berrien,et al.  Errors in perceiving the temporal order of auditory and visual stimuli. , 1942 .

[3]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[4]  D. Mackay Perceptual Stability of a Stroboscopically Lit Visual Field containing Self-Luminous Objects , 1958, Nature.

[5]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[6]  Jaj Jacques Roufs,et al.  Perception lag as a function of stimulus luminance , 1963 .

[7]  I. Hirsh,et al.  [SIMULTANEOUS CHARACTER AND SUCCESSION OF HETEROGENOUS STIMULI]. , 1964, L'annee psychologique.

[8]  P. Fraisse,et al.  Simultanéité et succession de stimuli hétérogènes , 1964 .

[9]  J. Rutschmann,et al.  Perception of Temporal Order of Stimuli Differing in Sense Mode and Simple Reaction Time , 1964, Perceptual and motor skills.

[10]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[11]  J. Gibson The Senses Considered As Perceptual Systems , 1967 .

[12]  A J Dinnerstein,et al.  Intermodal Perception of Temporal Order and Motor Skills: Effects of Age , 1968, Perceptual and motor skills.

[13]  A. Sanford Effects of Changes in the Intensity of White Noise on Simultaneity Judgements and Simple Reaction Time † , 1971 .

[14]  G. R. Engel,et al.  Visual-Auditory Distance Constancy , 1971, Nature.

[15]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[16]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[17]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[18]  J. Blauert Spatial Hearing: The Psychophysics of Human Sound Localization , 1983 .

[19]  Q. Summerfield,et al.  Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. , 1985, The Journal of the Acoustical Society of America.

[20]  Q. Summerfield Some preliminaries to a comprehensive account of audio-visual speech perception. , 1987 .

[21]  R. Campbell,et al.  Hearing by eye : the psychology of lip-reading , 1988 .

[22]  Felice L. Bedford,et al.  Constraints on learning new mappings between perceptual dimensions , 1989 .

[23]  D. Burke,et al.  Conduction velocities of muscle and cutaneous afferents in the upper and lower limbs of human subjects. , 1989, Brain : a journal of neurology.

[24]  Piotr Jaśkowski,et al.  Temporal-order judgments and reaction time for stimuli of different modalities , 1990, Psychological research.

[25]  R. D. Frey,et al.  Selective attention, event perception and the criterion of acceptability principle: evidence supporting and rejecting the doctrine of prior entry , 1990 .

[26]  L. Stelmach,et al.  Directed attention and perception of temporal order. , 1991, Journal of experimental psychology. Human perception and performance.

[27]  B. Stein,et al.  The Merging of the Senses , 1993 .

[28]  Romi Nijhawan,et al.  Motion extrapolation in catching , 1994, Nature.

[29]  Christian Abry,et al.  Auditory-visual spatial interaction and modularity. Commentaries. Author's response , 1994 .

[30]  R. Lynn,et al.  A secular decline in the strength of Spearman’s g in Japan , 1994 .

[31]  M. Radeau Auditory-visual spatial interaction and modularity , 1994, Current psychology of cognition = Cahiers de psychologie cognitive : CPC.

[32]  P. Gribble,et al.  Temporal constraints on the McGurk effect , 1996, Perception & psychophysics.

[33]  D. Lewkowicz Perception of auditory-visual temporal synchrony in human infants. , 1996, Journal of experimental psychology. Human perception and performance.

[34]  J. D. Mollon,et al.  Errors of judgement at Greenwich in 1796 , 1996, Nature.

[35]  S. McConnell,et al.  Virtual conferencing : Telepresence , 1997 .

[36]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[37]  Jüri Allik,et al.  Multiple Visual Latency , 1997 .

[38]  S. McConnell,et al.  Virtual conferencing , 1997 .

[39]  R. Nijhawan,et al.  Visual decomposition of colour through motion extrapolation , 1997, Nature.

[40]  J. Enns,et al.  Object Substitution: A New Form of Masking in Unattended Visual Locations , 1997 .

[41]  Jüri Allik,et al.  Multiple Visual Latency , 1998 .

[42]  G. Aschersleben,et al.  Automatic visual bias of perceived auditory location , 1998 .

[43]  R. Ulrich,et al.  Directed attention prolongs the perceived duration of a brief stimulus , 1998, Perception & psychophysics.

[44]  R. Welch Chapter 15 Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions , 1999 .

[45]  T. Bachmann,et al.  Cognitive contributions to the perception of spatial and temporal events (Advances in Psychology, Vol. 129) , 1999 .

[46]  P. Jaśkowski Chapter 11 Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations , 1999 .

[47]  P. Bertelson Chapter 14 Ventriloquism: A case of crossmodal perceptual grouping , 1999 .

[48]  R. Verleger,et al.  Attentional Bias toward Low-Intensity Stimuli: An Explanation for the Intensity Dissociation between Reaction Time and Temporal Order Judgment? , 2000, Consciousness and Cognition.

[49]  J. Enns,et al.  What’s new in visual masking? , 2000, Trends in Cognitive Sciences.

[50]  Heinrich H. Bülthoff,et al.  Touch can change visual slant perception , 2000, Nature Neuroscience.

[51]  C. Spence,et al.  Multisensory prior entry. , 2001, Journal of experimental psychology. General.

[52]  Shinsuke Shimojo,et al.  Beyond perceptual modality: Auditory effects on visual perception , 2001 .

[53]  R Fendrich,et al.  The temporal cross-capture of audition and vision , 2001, Perception & psychophysics.

[54]  C. Spence,et al.  Visual Prior Entry , 2001, Psychological science.

[55]  James V. Stone,et al.  When is now? Perception of simultaneity , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[56]  G. Recanzone,et al.  Temporal and spatial dependency of the ventriloquism effect , 2001, Neuroreport.

[57]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[58]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[59]  R. Nijhawan,et al.  Neural delays, visual motion and the flash-lag effect , 2002, Trends in Cognitive Sciences.

[60]  Yôiti Suzuki,et al.  Implicit estimation of sound-arrival time , 2003, Nature.

[61]  J. Lewald,et al.  Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. , 2003, Brain research. Cognitive brain research.

[62]  R. Baddeley,et al.  Multisensory temporal order judgments: When two locations are better than one , 2003, Perception & psychophysics.

[63]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[64]  D. Bavelier,et al.  Components of visual prior entry , 2003, Cognitive Psychology.

[65]  G. Aschersleben,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[66]  Charles Spence,et al.  Multisensory temporal order judgments: the role of hemispheric redundancy. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[67]  C. Spence,et al.  Audiovisual temporal order judgments , 2003, Experimental Brain Research.

[68]  N. Yakimoff,et al.  Mechanisms and asymmetries in visual perception of simultaneity and temporal order , 1986, Biological Cybernetics.

[69]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[70]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[71]  A. J. King,et al.  Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus , 2004, Experimental Brain Research.

[72]  J. Enns Object substitution and its relation to other forms of visual masking , 2004, Vision Research.

[73]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[74]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[75]  J. Lewald,et al.  Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception , 2004, Neuroscience Letters.

[76]  J. Vroomen,et al.  Temporal ventriloquism: sound modulates the flash-lag effect. , 2004, Journal of experimental psychology. Human perception and performance.

[77]  O. Neumann,et al.  Timing of perception and perception of time , 2004 .

[78]  L. Harris,et al.  Simultaneity Constancy , 2004, Perception.

[79]  L. Harris,et al.  Simultaneity constancy: detecting events with touch and vision , 2005, Experimental Brain Research.

[80]  Ernst Pöppel,et al.  Sensory integration within temporally neutral systems states: A hypothesis , 1990, Naturwissenschaften.

[81]  J. Vroomen,et al.  An event-related potential investigation of the time-course of temporal ventriloquism , 2005, Neuroreport.

[82]  Derek H. Arnold,et al.  Timing sight and sound , 2005, Vision Research.

[83]  Simon Carlile,et al.  Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[84]  Jean Vroomen,et al.  The role of spatial disparity and hemifields in audio-visual temporal order judgments , 2005, Experimental Brain Research.

[85]  C. Spence,et al.  Audiovisual prior entry , 2003, Neuroscience Letters.

[86]  C. Spence,et al.  CHAPTER 15 – Prior Entry , 2005 .

[87]  C. Spence,et al.  Audio-visual simultaneity judgments , 2005, Perception & psychophysics.

[88]  Charles Spence,et al.  Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. , 2005, Brain research. Cognitive brain research.

[89]  C. Spence,et al.  Audiotactile temporal order judgments. , 2005, Acta psychologica.

[90]  Waka Fujisaki,et al.  Temporal frequency characteristics of synchrony–asynchrony discrimination of audio-visual signals , 2005, Experimental Brain Research.

[91]  D. Pisoni,et al.  Auditory-visual speech perception and synchrony detection for speech and nonspeech signals. , 2006, The Journal of the Acoustical Society of America.

[92]  P. Montague,et al.  Motor-Sensory Recalibration Leads to an Illusory Reversal of Action and Sensation , 2006, Neuron.

[93]  Jeffery A. Jones,et al.  Multisensory integration of speech signals: the relationship between space and time , 2006, Experimental Brain Research.

[94]  C. Spence,et al.  Audiovisual synchrony perception for music, speech, and object actions , 2006, Brain Research.

[95]  David Whitaker,et al.  Adaptation minimizes distance-related audiovisual delays. , 2007, Journal of vision.

[96]  Charles Spence,et al.  Temporal recalibration during asynchronous audiovisual speech perception , 2007, Experimental Brain Research.

[97]  C. Spence Audiovisual multisensory integration , 2007 .

[98]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[99]  Stephan Getzmann,et al.  The Effect of Brief Auditory Stimuli on Visual Apparent Motion , 2007, Perception.

[100]  Perceived timing across modalities , 2007 .

[101]  Jean Vroomen,et al.  Neural Correlates of Multisensory Integration of Ecologically Valid Audiovisual Events , 2007, Journal of Cognitive Neuroscience.

[102]  Charles Spence,et al.  Adaptation to audiotactile asynchrony , 2007, Neuroscience Letters.

[103]  L. Harris,et al.  The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity , 2008, Experimental Brain Research.

[104]  Jean Vroomen,et al.  No effect of auditory–visual spatial disparity on temporal recalibration , 2007, Experimental Brain Research.

[105]  C. Spence,et al.  Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli , 2007, Perception & psychophysics.

[106]  Gerald Westheimer,et al.  Grouping of contextual elements that affect vernier thresholds. , 2007, Journal of vision.

[107]  Salvador Soto-Faraco,et al.  Conscious access to the unisensory components of a cross-modal illusion , 2007, Neuroreport.

[108]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[109]  Asif A Ghazanfar,et al.  Facilitation of multisensory integration by the "unity effect" reveals that speech is special. , 2008, Journal of vision.

[110]  D. Whitaker,et al.  Recalibration of perceived time across sensory modalities , 2008, Experimental Brain Research.

[111]  J. Vroomen,et al.  Temporal recalibration to tactile–visual asynchronous stimuli , 2008, Neuroscience Letters.

[112]  Jon Driver,et al.  Direction of Visual Apparent Motion Driven Solely by Timing of a Static Sound , 2008, Current Biology.

[113]  J. Juola,et al.  Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type , 2008, Perception & psychophysics.

[114]  J. Vroomen,et al.  Tactile—visual temporal ventriloquism: No effect of spatial disparity , 2008, Perception & psychophysics.

[115]  van Rlj Rob Eijk,et al.  Audio-visual synchrony perception , 2008 .

[116]  C. Spence,et al.  Evaluating the influence of the 'unity assumption' on the temporal perception of realistic audiovisual stimuli. , 2008, Acta psychologica.

[117]  Jeroen J. A. van Boxtel,et al.  Retinotopic and non-retinotopic stimulus encoding in binocular rivalry and the involvement of feedback. , 2008, Journal of vision.

[118]  Frans A. J. Verstraten,et al.  Matching Auditory and Visual Signals: Is Sensory Modality Just Another Feature? , 2008, Perception.

[119]  Katsumi Watanabe,et al.  Realignment of temporal simultaneity between vision and touch , 2008, Neuroreport.

[120]  S. Nishida,et al.  Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception , 2009, Experimental Brain Research.

[121]  Jean Vroomen,et al.  Sounds change four-dot masking. , 2009, Acta psychologica.

[122]  S. Soto-Faraco,et al.  Deconstructing the McGurk-MacDonald illusion. , 2009, Journal of experimental psychology. Human perception and performance.

[123]  Steven van de Par,et al.  Temporal interval discrimination thresholds depend on perceived synchrony for audio-visual stimulus pairs. , 2009, Journal of experimental psychology. Human perception and performance.

[124]  Massimiliano Di Luca,et al.  Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. , 2009, Journal of vision.

[125]  J. Mollon,et al.  Latency characteristics of the short-wavelength-sensitive cones and their associated pathways. , 2009, Journal of vision.

[126]  C. Spence,et al.  Adaptation to audiovisual asynchrony modulates the speeded detection of sound , 2009, Proceedings of the National Academy of Sciences.

[127]  F. Pollick,et al.  When knowing can replace seeing in audiovisual integration of actions , 2009, Cognition.

[128]  H. Imai,et al.  Temporal Recalibration in Audio-Visual Speech Integration Using a Simultaneity Judgment Task and the McGurk Identification Task , 2009 .

[129]  Shuichi Sakamoto,et al.  Visual speech improves the intelligibility of time-expanded auditory speech , 2009, Neuroreport.

[130]  Measuring Video Quality in Videoconferencing Systems , 2009 .

[131]  Jean Vroomen,et al.  Visual Anticipatory Information Modulates Multisensory Interactions of Artificial Audiovisual Stimuli , 2010, Journal of Cognitive Neuroscience.

[132]  S. Kitazawa,et al.  Bayesian Calibration of Simultaneity in Audiovisual Temporal Order Judgments , 2012, PloS one.

[133]  C. Spence,et al.  Perceptual effects of cross-modal stimulation , 2022 .