Judging Relative Onsets and Offsets of Audiovisual Events

This study assesses the fidelity with which people can make temporal order judgments (TOJ) between auditory and visual onsets and offsets. Using an adaptive staircase task administered to a large sample of young adults, we find that the ability to judge temporal order varies widely among people, with notable difficulty created when auditory events closely follow visual events. Those findings are interpretable within the context of an independent channels model. Visual onsets and offsets can be difficult to localize in time when they occur within the temporal neighborhood of sound onsets or offsets.

[1]  C Miniussi,et al.  Orienting attention in time. , 2001, Frontiers in bioscience : a journal and virtual library.

[2]  Steven van de Par,et al.  Temporal interval discrimination thresholds depend on perceived synchrony for audio-visual stimulus pairs. , 2009, Journal of experimental psychology. Human perception and performance.

[3]  Ernest R. Hilgard,et al.  Reinforcement and Inhibition of Eyelid Reflexes , 1933 .

[4]  W. Newsome,et al.  Motion selectivity in macaque visual cortex. I. Mechanisms of direction and speed selectivity in extrastriate area MT. , 1986, Journal of neurophysiology.

[5]  D. Shore,et al.  Exploring the Role of the Amplitude Envelope in Duration Estimation , 2014, Perception.

[6]  Cyril R Pernet,et al.  Beyond differences in means: robust graphical methods to compare two groups in neuroscience , 2017, bioRxiv.

[7]  A. Leventhal,et al.  Signal timing across the macaque visual system. , 1998, Journal of neurophysiology.

[8]  L. Rüschendorf,et al.  On the Perception of Time , 2009, Gerontology.

[9]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[10]  M. Meredith,et al.  On the neuronal basis for multisensory convergence: a brief overview. , 2002, Brain research. Cognitive brain research.

[11]  R. Blake,et al.  Audio-visual interactions uniquely contribute to resolution of visual conflict in people possessing absolute pitch , 2017, PloS one.

[12]  David C. Burr,et al.  Cue Combination Within a Bayesian Framework , 2019, Multisensory Processes.

[13]  Virginie van Wassenhove,et al.  Time Order as Psychological Bias , 2017, Psychological science.

[14]  D. Bavelier,et al.  Components of visual prior entry , 2003, Cognitive Psychology.

[15]  M. Wallace,et al.  Event Related Potentials Index Rapid Recalibration to Audiovisual Temporal Asynchrony , 2017, Front. Integr. Neurosci..

[16]  M. Wallace,et al.  Multisensory Integration as a Window into Orderly and Disrupted Cognition and Communication. , 2020, Annual review of psychology.

[17]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[18]  R. L. Knoll,et al.  The Perception of Temporal Order: Fundamental Issues and a General Model , 1973 .

[19]  Mowbray Gh,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959 .

[20]  Charles Spence,et al.  Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. , 2005, Brain research. Cognitive brain research.

[21]  P. Bertelson,et al.  Recalibration of temporal order perception by exposure to audio-visual asynchrony. , 2004, Brain research. Cognitive brain research.

[22]  I. Hirsh,et al.  Perceived order in different sense modalities. , 1961, Journal of experimental psychology.

[23]  Miguel A. García-Pérez,et al.  Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines , 2013, Behavior research methods.

[24]  Andreas Bartels,et al.  Naturalistic Stimulus Structure Determines the Integration of Audiovisual Looming Signals in Binocular Rivalry , 2013, PloS one.

[25]  W. Meck,et al.  Temporal cognition: Connecting subjective time to perception, attention, and memory. , 2016, Psychological bulletin.

[26]  R. Blake,et al.  Congruent audio-visual stimulation during adaptation modulates the subsequently experienced visual motion aftereffect , 2019, Scientific Reports.

[27]  Gian Domenico Iannetti,et al.  An Action Field Theory of Peripersonal Space , 2018, Trends in Cognitive Sciences.

[28]  R. Heath.  Response Time and Temporal Order Judgement in Vision , 1984 .

[29]  James V. Stone,et al.  When is now? Perception of simultaneity , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[30]  G. Recanzone.  Auditory influences on visual temporal rate perception. , 2003, Journal of neurophysiology.

[31]  D W Massaro,et al.  Perception of asynchronous and conflicting visual and auditory speech. , 1996, The Journal of the Acoustical Society of America.

[32]  Shin'ya Nishida,et al.  The sliding window of audio-visual simultaneity. , 2009, Journal of vision.

[33]  F. K. Berrien,et al.  Errors in perceiving the temporal order of auditory and visual stimuli. , 1942 .

[34]  E. S. Robinson.  A History of Experimental Psychology , 1930 .

[35]  D L Kohfeld,et al.  Simple reaction time as a function of stimulus intensity in decibels of light and sound. , 1971, Journal of experimental psychology.

[36]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.

[37]  G H MOWBRAY,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959, The American journal of psychology.

[38]  E. Freeman,et al.  Correlation of Individual Differences in Audiovisual Asynchrony Across Stimuli and Tasks: New Constraints on Temporal Renormalization Theory , 2018, Journal of experimental psychology. Human perception and performance.

[39]  Lars T. Boenke,et al.  Flash illusions induced by visual, auditory, and audiovisual stimuli. , 2013, Journal of vision.

[40]  J. Wearden,et al.  Why “Sounds Are Judged Longer than Lights”: Application of a Model of the Internal Clock in Humans , 1998, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[41]  Thomas V. Wiecki,et al.  The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models , 2018, Psychonomic Bulletin & Review.

[42]  P. Lennie.  The physiological basis of variations in visual latency , 1981, Vision Research.

[43]  R. Klein,et al.  Saccadic Performance as a Function of the Presence and Disappearance of Auditory and Visual Fixation Stimuli , 1999, Journal of Cognitive Neuroscience.

[44]  M. Morgan,et al.  Observers can voluntarily shift their psychometric functions without losing sensitivity , 2011, Attention, Perception, & Psychophysics.

[45]  J. Bruner,et al.  On the perception of incongruity; a paradigm. , 1949, Journal of personality.

[46]  C. Spence,et al.  Audiovisual temporal order judgments , 2003, Experimental Brain Research.

[47]  Simon Schaffer,et al.  Astronomers Mark Time: Discipline and the Personal Equation , 1988, Science in Context.

[48]  J. Vroomen,et al.  Intersensory binding across space and time: A tutorial review , 2013, Attention, Perception, & Psychophysics.

[49]  C. Camalier,et al.  Neural latencies across auditory cortex of macaque support a dorsal stream supramodal timing advantage in primates , 2012, Proceedings of the National Academy of Sciences.

[50]  D. Burr,et al.  Combining visual and auditory information. , 2006, Progress in brain research.

[51]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[52]  Adrian K. C. Lee,et al.  Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms , 2016, Trends in Neurosciences.

[53]  N. Bolognini,et al.  “Acoustical vision” of below threshold stimuli: interaction among spatially converging audiovisual inputs , 2004, Experimental Brain Research.

[54]  J. Vroomen,et al.  Perception of intersensory synchrony: A tutorial review , 2010, Attention, perception & psychophysics.

[55]  P. Fraisse.  Perception and estimation of time. , 1984, Annual review of psychology.

[56]  Alberta Ipser,et al.  Sight and sound persistently out of synch: stable individual differences in audiovisual synchronisation revealed by implicit measures of lip-voice integration , 2017, Scientific Reports.

[57]  Anton Olsson-Collentine,et al.  The Prevalence of Marginally Significant Results in Psychology Over Time , 2019, Psychological science.

[58]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[59]  Robert Turner,et al.  Resolving multisensory and attentional influences across cortical depth in sensory cortices , 2019, bioRxiv.

[60]  A. Gorea,et al.  Temporal order judgment and simple reaction times: evidence for a common processing system. , 2007, Journal of vision.

[61]  Randolph Blake,et al.  Hearing What the Eyes See , 2005, Psychological science.

[62]  A. Diederich,et al.  The time window of multisensory integration: relating reaction times and judgments of temporal order. , 2015, Psychological review.

[63]  C. Spence,et al.  Visual Prior Entry , 2001, Psychological science.

[64]  Neil W. Roach,et al.  Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration , 2006, Proceedings of the Royal Society B: Biological Sciences.

[65]  Derek Powell,et al.  Marginally Significant Effects as Evidence for Hypotheses , 2016, Psychological science.

[66]  R. Blake,et al.  Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music , 2015, Proceedings of the National Academy of Sciences.

[67]  S. Nishida,et al.  Recalibration of audiovisual simultaneity , 2004, Nature Neuroscience.

[68]  V. Walsh,et al.  Time and the brain: neurorelativity The chronoarchitecture of the brain from the neuronal rather than the observer's perspective , 2013, Trends in Cognitive Sciences.

[69]  Richard P. Heitz,et al.  Neurally constrained modeling of perceptual decision making. , 2010, Psychological review.

[70]  M. Wallace,et al.  The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities , 2014, Neuropsychologia.

[71]  Alfred B. Kristofferson,et al.  Successiveness discrimination: Two models , 1974 .

[72]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[73]  Philip J. B. Jackson,et al.  Limits of Perceived Audio-Visual Spatial Coherence as Defined by Reaction Time Measurements , 2019, Front. Neurosci..

[74]  Michael J Kahana,et al.  Assessing recognition memory using confidence ratings and response times , 2016, Royal Society Open Science.

[75]  Clara Casco,et al.  Audiovisual bounce-inducing effect: attention alone does not explain why the discs are bouncing. , 2009, Journal of experimental psychology. Human perception and performance.

[76]  E N Pugh,et al.  A quantitative account of the activation steps involved in phototransduction in amphibian photoreceptors. , 1992, The Journal of physiology.

[77]  P. Reuter-Lorenz,et al.  Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. , 1994, Journal of experimental psychology. Human perception and performance.

[78]  M. Naumer,et al.  Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration , 2008, Brain Research.

[79]  J. Gross,et al.  Being First Matters: Topographical Representational Similarity Analysis of ERP Signals Reveals Separate Networks for Audiovisual Temporal Binding Depending on the Leading Sense , 2017, The Journal of Neuroscience.

[80]  M. García-Pérez,et al.  On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model , 2012, Psychonomic bulletin & review.

[81]  R. Fay,et al.  Multisensory Processes: The Auditory Perspective , 2019, Springer Handbook of Auditory Research.

[82]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[83]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[84]  R. Schlauch,et al.  Duration discrimination and subjective duration for ramped and damped sounds. , 2001, The Journal of the Acoustical Society of America.

[85]  M. Grassi,et al.  The subjective duration of ramped and damped sounds , 2006, Perception & psychophysics.

[86]  W. Singer,et al.  Auditory Motion Capturing Ambiguous Visual Motion , 2012, Front. Psychology.

[87]  Dora E Angelaki,et al.  Dissecting neural circuits for multisensory integration and crossmodal processing , 2015, Philosophical Transactions of the Royal Society B: Biological Sciences.

[88]  G. Recanzone.  Response profiles of auditory cortical neurons to tones and noise in behaving macaque monkeys , 2000, Hearing Research.

[89]  F. McConnel,et al.  The Auditory Periphery , 1975 .

[90]  Ryota Kanai,et al.  Dissociable Neuroanatomical Correlates of Subsecond and Suprasecond Time Perception , 2014, Journal of Cognitive Neuroscience.

[91]  Ryota Kanai,et al.  What contributes to individual differences in brain structure? , 2014, Front. Hum. Neurosci..