Microsaccadic Eye Movements but not Pupillary Dilation Response Characterizes the Crossmodal Freezing Effect

Abstract In typical spatial orienting tasks, the perception of crossmodal (e.g., audiovisual) stimuli evokes greater pupil dilation and microsaccade inhibition than unisensory stimuli (e.g., visual). The characteristic pupil dilation and microsaccade inhibition has been observed in response to “salient” events/stimuli. Although the “saliency” account is appealing in the spatial domain, whether this occurs in the temporal context remains largely unknown. Here, in a brief temporal scale (within 1 s) and with the working mechanism of involuntary temporal attention, we investigated how eye metric characteristics reflect the temporal dynamics of perceptual organization, with and without multisensory integration. We adopted the crossmodal freezing paradigm using the classical Ternus apparent motion. Results showed that synchronous beeps biased the perceptual report for group motion and triggered the prolonged sound-induced oculomotor inhibition (OMI), whereas the sound-induced OMI was not obvious in a crossmodal task-free scenario (visual localization without audiovisual integration). A general pupil dilation response was observed in the presence of sounds in both visual Ternus motion categorization and visual localization tasks. This study provides the first empirical account of crossmodal integration by capturing microsaccades within a brief temporal scale; OMI but not pupillary dilation response characterizes task-specific audiovisual integration (shown by the crossmodal freezing effect).

[1]  S. Shimojo,et al.  When Sound Affects Vision: Effects of Auditory Grouping on Visual Motion Perception , 2001, Psychological science.

[2]  Rodrigo M. Braga,et al.  Auditory and visual connectivity gradients in frontoparietal cortex , 2016, Human brain mapping.

[3]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[4]  M. Rolfs Microsaccades: Small steps on a long way , 2009, Vision Research.

[5]  Erich Schröger,et al.  Microsaccadic Responses Indicate Fast Categorization of Sounds: A Novel Approach to Study Auditory Cognition , 2014, The Journal of Neuroscience.

[6]  Shin'ya Nishida,et al.  Direction of visual apparent motion driven by perceptual organization of cross-modal signals. , 2013, Journal of vision.

[7]  Kazuo Fujita,et al.  Differential motion processing between species facing Ternus–Pikler display: Non-retinotopic humans versus retinotopic pigeons , 2014, Vision Research.

[8]  Stephen Grossberg,et al.  A neural architecture for visual motion perception: group and element apparent motion , 1989, International 1989 Joint Conference on Neural Networks.

[9]  Ralf Engbert,et al.  Microsaccades are triggered by low retinal image slip. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[10]  U. Polat,et al.  ADHD subjects fail to suppress eye blinks and microsaccades while anticipating visual stimuli but recover with medication , 2014, Vision Research.

[11]  J M Findlay,et al.  Direction perception and human fixation eye movements. , 1974, Vision research.

[12]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[13]  Jillian H. Fecteau,et al.  The behavioural and neurophysiological modulation of microsaccades in monkeys , 2009 .

[14]  Ziad M. Hafed,et al.  On the Dissociation between Microsaccade Rate and Direction after Peripheral Cues: Microsaccadic Inhibition Revisited , 2013, The Journal of Neuroscience.

[15]  S. Furukawa,et al.  Human Pupillary Dilation Response to Deviant Auditory Stimuli: Effects of Stimulus Properties and Voluntary Attention , 2016, Front. Neurosci..

[16]  S. Yasuda,et al.  Separation process of two-phase fluids , 2005, J. Vis..

[17]  G. Pari,et al.  Disruption of pupil size modulation correlates with voluntary motor preparation deficits in Parkinson's disease , 2016, Neuropsychologia.

[18]  Zhuanghua Shi,et al.  Influences of intra- and crossmodal grouping on visual and tactile Ternus apparent motion , 2010, Brain Research.

[19]  Maria Concetta Morrone,et al.  "Non-retinotopic processing" in Ternus motion displays modeled by spatiotemporal filters. , 2012, Journal of vision.

[20]  A. Mizuno,et al.  A change of the leading player in flow Visualization technique , 2006, J. Vis..

[21]  Xoana G. Troncoso,et al.  Microsaccades drive illusory motion in the Enigma illusion , 2008, Proceedings of the National Academy of Sciences.

[22]  Francisco M. Costela,et al.  Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes , 2014, The European journal of neuroscience.

[23]  Joseph Krummenacher,et al.  A (fascinating) litmus test for human retino- vs. non-retinotopic processing. , 2009, Journal of vision.

[24]  Alexandre Pouget,et al.  Neural Correlates of Optimal Multisensory Decision Making under Time-Varying Reliabilities with an Invariant Linear Probabilistic Population Code , 2019, Neuron.

[25]  S. Yantis,et al.  Perceptual grouping in space and time: Evidence from the Ternus display , 1997, Perception & psychophysics.

[26]  Marisa Carrasco,et al.  Directing Voluntary Temporal Attention Increases Fixational Stability , 2018, The Journal of Neuroscience.

[27]  Masahiro Takei,et al.  Human resource development and visualization , 2009, J. Vis..

[28]  Christian N L Olivers,et al.  Oculomotor measures reveal the temporal dynamics of preparing for search. , 2017, Progress in brain research.

[29]  Zhuanghua Shi,et al.  Interaction of Perceptual Grouping and Crossmodal Temporal Capture in Tactile Apparent-Motion , 2011, PloS one.

[30]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[31]  Ziad M. Hafed,et al.  Microsaccadic Suppression of Visual Bursts in the Primate Superior Colliculus , 2010, Journal of Neuroscience.

[32]  J. Wearden,et al.  Why “Sounds Are Judged Longer than Lights”: Application of a Model of the Internal Clock in Humans , 1998, The Quarterly journal of experimental psychology. B, Comparative and physiological psychology.

[33]  S. Martinez-Conde Fixational eye movements in normal and pathological vision. , 2006, Progress in brain research.

[34]  Zhuanghua Shi,et al.  What you see depends on what you hear: temporal averaging and crossmodal integration , 2018, bioRxiv.

[35]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[36]  M T Wallace,et al.  Development of Multisensory Neurons and Multisensory Integration in Cat Superior Colliculus , 1997, The Journal of Neuroscience.

[37]  Xoana G. Troncoso,et al.  Microsaccades Counteract Visual Fading during Fixation , 2005, Neuron.

[38]  Alexander Pastukhov,et al.  Spatial and temporal attention revealed by microsaccades , 2013, Vision Research.

[39]  Ralf Engbert,et al.  Crossmodal coupling of oculomotor control and spatial attention in vision and audition , 2005, Experimental Brain Research.

[40]  J. Braun,et al.  Rare but precious: Microsaccades are highly informative about attentional allocation , 2010, Vision Research.

[41]  R. Kliegl,et al.  Microsaccade-induced prolongation of saccade latencies depends on microsaccade amplitude , 2008 .

[42]  Krzysztof Krejtz,et al.  Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze , 2018, PloS one.

[43]  Marco Boi,et al.  Motion and tilt aftereffects occur largely in retinal, not in object, coordinates in the Ternus-Pikler display. , 2011, Journal of vision.

[44]  J. Vroomen,et al.  Intersensory binding across space and time: A tutorial review , 2013, Attention, Perception, & Psychophysics.

[45]  Ralf Engbert,et al.  Microsaccades uncover the orientation of covert attention , 2003, Vision Research.

[46]  M. Turatto,et al.  Are you ready? I can tell by looking at your microsaccades , 2006, Neuroreport.

[47]  Shigeto Furukawa,et al.  Correspondences among pupillary dilation response, subjective salience of sounds, and loudness , 2015, Psychonomic Bulletin & Review.

[48]  Marisa Carrasco,et al.  Oculomotor inhibition reflects temporal expectations , 2019, NeuroImage.

[49]  Jon Driver,et al.  Direction of Visual Apparent Motion Driven Solely by Timing of a Static Sound , 2008, Current Biology.

[50]  Ziad M. Hafed,et al.  Modulation of Microsaccades in Monkey during a Covert Visual Attention Task , 2011, The Journal of Neuroscience.

[51]  D. Munoz,et al.  On the importance of the transient visual response in the superior colliculus , 2008, Current Opinion in Neurobiology.

[52]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[53]  Brian J. White,et al.  Microstimulation of the Monkey Superior Colliculus Induces Pupil Dilation Without Evoking Saccades , 2012, The Journal of Neuroscience.

[54]  W. Wildman,et al.  Theoretical Neuroscience , 2014 .

[55]  Ralf Engbert,et al.  Modeling the control of fixational eye movements with neurophysiological delays. , 2007, Physical review letters.

[56]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[57]  Gijs Plomp,et al.  Retinotopic encoding of the Ternus-Pikler display reflected in the early visual areas , 2016, Journal of vision.

[58]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[59]  Laurence R Harris,et al.  Multimodal Ternus: Visual, Tactile, and Visuo — Tactile Grouping in Apparent Motion , 2007, Perception.

[60]  Haluk Öğmen,et al.  Attention and non-retinotopic feature integration. , 2010, Journal of vision.

[61]  Mauro Ursino,et al.  Multisensory integration in the superior colliculus: a neural network model , 2009, Journal of Computational Neuroscience.

[62]  D. Munoz,et al.  Pupil size reveals preparatory processes in the generation of pro‐saccades and anti‐saccades , 2015, The European journal of neuroscience.

[63]  Patrick Cavanagh,et al.  Motion correspondence in the Ternus display shows feature bias in spatiotopic coordinates. , 2012, Journal of vision.

[64]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.

[65]  Emiliano Macaluso,et al.  Auditory temporal expectations modulate activity in visual cortex , 2010, NeuroImage.

[66]  Ralf Engbert,et al.  Toward a model of microsaccade generation: the case of microsaccadic inhibition. , 2008, Journal of vision.

[67]  S. Martinez-Conde,et al.  The impact of microsaccades on vision: towards a unified theory of saccadic function , 2013, Nature Reviews Neuroscience.

[68]  de Gelder Sound Enhances Visual Perception: Cross-Modal Effects of Auditory Organization on Vision , 2001 .

[69]  Wolfgang Nitsche,et al.  Infrared based visualization of wall shear stress distributions with a high temporal and spatial resolution , 2007, J. Vis..

[70]  Barbara G Shinn-Cunningham,et al.  Cerebral Cortex doi:10.1093/cercor/bhs359 Auditory Spatial Attention Representations in the Human Cerebral Cortex , 2012 .

[71]  Ziad M. Hafed,et al.  Similarity of superior colliculus involvement in microsaccade and saccade generation. , 2012, Journal of neurophysiology.

[72]  Marisa Carrasco,et al.  Oculomotor freezing reflects tactile temporal expectation and aids tactile perception , 2020, Nature Communications.

[73]  Xoana G. Troncoso,et al.  Microsaccades: a neurophysiological analysis , 2009, Trends in Neurosciences.

[74]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[75]  Zhuanghua Shi,et al.  Auditory temporal modulation of the visual Ternus effect: the influence of time interval , 2010, Experimental Brain Research.

[76]  Xoana G. Troncoso,et al.  Saccades and microsaccades during visual fixation, exploration, and search: foundations for a common saccadic generator. , 2008, Journal of vision.

[77]  Mark Mon-Williams,et al.  An illusion that avoids focus , 1996, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[78]  Peter De Weerd,et al.  The Attentional Blink is Related to the Microsaccade Rate Signature. , 2019, Cerebral cortex.

[79]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[80]  B. L. Zuber,et al.  Saccadic suppression: elevation of visual threshold associated with saccadic eye movements. , 1966, Experimental neurology.

[81]  Gunnar Blohm,et al.  Multisensory integration in orienting behavior: Pupil size, microsaccades, and saccades , 2017, Biological Psychology.

[82]  Stephen J. Johnston,et al.  Fixation Region Overlap: A quantitative method for the analysis of fixational eye movement patterns , 2009 .

[83]  Haluk Ögmen,et al.  Reference-frames in vision: Contributions of attentional tracking to nonretinotopic perception in the Ternus-Pikler display. , 2019, Journal of vision.

[84]  Michael J. Crosse,et al.  Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term Crossmodal Temporal Integration , 2016, The Journal of Neuroscience.

[85]  G. W. Beeler,et al.  Visual threshold changes resulting from spontaneous saccadic eye movements. , 1967, Vision research.

[86]  Corrigendum: Prestimulus Inhibition of Saccades in Adults With and Without Attention-Deficit/Hyperactivity Disorder as an Index of Temporal Expectations , 2017, Psychological science.

[87]  M. Wallace,et al.  Multisensory integration in the superior colliculus of the alert cat. , 1998, Journal of neurophysiology.

[88]  Laurent Itti,et al.  Transient Pupil Response Is Modulated by Contrast-Based Saliency , 2014, The Journal of Neuroscience.

[89]  Robin Walker,et al.  What determines the direction of microsaccades , 2010 .

[90]  J. Ternus Experimentelle Untersuchungen über phänomenale Identität , 1926 .

[91]  Andrew J. Anderson,et al.  The influence of retinal image motion on the perceptual grouping of temporally asynchronous stimuli , 2019, Journal of vision.

[92]  N. Holmes The Principle of Inverse Effectiveness in Multisensory Integration: Some Statistical Considerations , 2009, Brain Topography.

[93]  Daniel P. Newman,et al.  Antagonistic Interactions Between Microsaccades and Evidence Accumulation Processes During Decision Formation , 2018, The Journal of Neuroscience.

[94]  Ramis Örlü,et al.  Flow visualization of an oblique impinging jet: vortices like it downhill, not uphill , 2016, J. Vis..

[95]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[96]  L. Stark,et al.  The main sequence, a tool for studying human eye movements , 1975 .