Temporal auditory capture does not affect the time course of saccadic mislocalization of visual stimuli.

Irrelevant sounds can "capture" visual stimuli to change their apparent timing, a phenomenon sometimes termed "temporal ventriloquism". Here we ask whether this auditory capture can alter the time course of spatial mislocalization of visual stimuli during saccades. We first show that during saccades, sounds affect the apparent timing of visual flashes, even more strongly than during fixation. However, this capture does not affect the dynamics of perisaccadic visual distortions. Sounds presented 50 ms before or after a visual bar (that change perceived timing of the bars by more than 40 ms) had no measurable effect on the time courses of spatial mislocalization of the bars, in four subjects. Control studies showed that with barely visible, low-contrast stimuli, leading, but not trailing, sounds can have a small effect on mislocalization, most likely attributable to attentional effects rather than auditory capture. These findings support previous studies showing that integration of multisensory information occurs at a relatively late stage of sensory processing, after visual representations have undergone the distortions induced by saccades.

[1]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[2]  Mowbray Gh,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959 .

[3]  James J. Clark,et al.  Data Fusion for Sensory Information Processing Systems , 1990 .

[4]  L. Matin,et al.  Visual Perception of Direction for Stimuli Flashed During Voluntary Saccadic Eye Movements , 1965, Science.

[5]  Jean Vroomen,et al.  Neural correlates of audiovisual motion capture , 2009, Experimental Brain Research.

[6]  D. H. Warren,et al.  The role of visual-auditory “compellingness” in the ventriloquism effect: Implications for transitivity among the spatial senses , 1981, Perception & psychophysics.

[7]  Jess Hartcher-O’Brien,et al.  Temporal Ventriloquism: Perceptual shifts forwards and backwards in time predicted by the maximum likelihood model , 2007 .

[8]  Eli Brenner,et al.  Temporal information can influence spatial localization. , 2009, Journal of neurophysiology.

[9]  D. Burr,et al.  Changes in visual perception at the time of saccades , 2001, Trends in Neurosciences.

[10]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[11]  T SHIPLEY,et al.  Auditory Flutter-Driving of Visual Flicker , 1964, Science.

[12]  D. Burr,et al.  The Ventriloquist Effect Results from Near-Optimal Bimodal Integration , 2004, Current Biology.

[13]  Tracey D. Berger,et al.  Flicker flutter: is an illusory event as good as the real thing? , 2003, Journal of vision.

[14]  A. Watson.  Probability summation over time , 1979, Vision Research.

[15]  M. Keeling,et al.  Involuntary orienting to sound improves visual perception , 2022 .

[16]  S. Mateeff,et al.  Dynamic Visual Capture: Apparent Auditory Motion Induced by a Moving Visual Target , 1985, Perception.

[17]  A. Kingstone,et al.  Auditory capture of vision: examining temporal ventriloquism. , 2003, Brain research. Cognitive brain research.

[18]  G. Aschersleben,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[19]  E. Titchener.  Scientific Books: Lectures on the Elementary Psychology of Feeling and Attention , 1909 .

[20]  M. Concetta Morrone,et al.  Apparent Position of Visual Targets during Real and Simulated Saccadic Eye Movements , 1997, The Journal of Neuroscience.

[21]  S. Inati,et al.  Eye Position Influences Auditory Responses in Primate Inferior Colliculus , 2001, Neuron.

[22]  Robert B. Welch,et al.  Contributions of audition and vision to temporal rate perception , 1986, Perception & psychophysics.

[23]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[24]  S. Sterbing-D’Angelo,et al.  Behavioral/systems/cognitive Multisensory Space Representations in the Macaque Ventral Intraparietal Area , 2022 .

[25]  M Concetta Morrone,et al.  Saccadic eye movements cause compression of time as well as space , 2005, Nature Neuroscience.

[26]  Guido Marco Cicchini,et al.  Spatiotemporal Distortions of Visual Perception at the Time of Saccades , 2009, The Journal of Neuroscience.

[27]  H. Honda.  Perceptual localization of visual stimuli flashed during saccades , 1989, Perception & psychophysics.

[28]  Charles Spence,et al.  ‘When Birds of a Feather Flock Together’: Synesthetic Correspondences Modulate Audiovisual Integration in Non-Synesthetes , 2009, PloS one.

[29]  P. Verghese,et al.  Combining speed information across space , 1995, Vision Research.

[30]  Donatella Spinelli,et al.  Electrophysiological evidence for an early attentional mechanism in visual processing in humans , 1999, Vision Research.

[31]  G. Recanzone.  Auditory influences on visual temporal rate perception. , 2003, Journal of neurophysiology.

[32]  M. Morrone,et al.  Extraretinal Control of Saccadic Suppression , 2000, The Journal of Neuroscience.

[33]  D. Burr,et al.  Auditory dominance over vision in the perception of interval duration , 2009, Experimental Brain Research.

[34]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[35]  J. Groh,et al.  Eye Position Affects Activity in Primary Auditory Cortex of Primates , 2003, Current Biology.

[36]  A. Pouget,et al.  Efficient computation and cue integration with noisy population codes , 2001, Nature Neuroscience.

[37]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[38]  G H MOWBRAY,et al.  On discriminating the rate of visual flicker and auditory flutter. , 1959, The American journal of psychology.

[39]  C. Spence,et al.  Cross-modal links in spatial attention. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[40]  R Fendrich,et al.  The temporal cross-capture of audition and vision , 2001, Perception & psychophysics.

[41]  Derek H. Arnold,et al.  Spatially Localized Distortions of Event Time , 2006, Current Biology.

[42]  Y. Cohen,et al.  Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. , 2005, Journal of neurophysiology.

[43]  Paul Bertelson,et al.  Temporal ventriloquism: crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[44]  Konrad Paul Kording,et al.  Sensory Cue Integration , 2011 .

[45]  David C. Burr,et al.  Compression of visual space before saccades , 1997, Nature.

[46]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[47]  M Concetta Morrone,et al.  Neural mechanisms for timing visual events are spatially selective in real-world coordinates , 2007, Nature Neuroscience.

[48]  Zoubin Ghahramani,et al.  Computation and psychophysics of sensorimotor integration , 1996 .

[49]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[50]  M. Radeau.  Ventriloquism against audio-visual speech: or, where Japanese-speaking barn owls might help , 1994, Current psychology of cognition = Cahiers de psychologie cognitive : CPC.

[51]  David Burr,et al.  Time Perception: Space–Time in the Brain , 2006, Current Biology.

[52]  D. Burr,et al.  Keeping vision stable: rapid updating of spatiotopic receptive fields may cause relativistic-like effects , 2010 .

[53]  David C Burr,et al.  Fusion of Visual and Auditory Stimuli during Saccades: A Bayesian Explanation for Perisaccadic Distortions , 2007, The Journal of Neuroscience.

[54]  Antigona Martínez,et al.  Source analysis of event-related cortical activity during visuo-spatial attention. , 2003, Cerebral cortex.