REMoDNaV: Robust Eye-Movement Classification for Dynamic Stimulation

Abstract Tracking of eye movements is an established measurement for many types of experimental paradigms. More complex and more prolonged visual stimuli have made algorithmic approaches to eye movement event classification the most pragmatic option. A recent analysis revealed that many current algorithms are lackluster when it comes to data from viewing dynamic stimuli such as video sequences. Here we present an event classification algorithm—built on an existing velocity-based approach—that is suitable for both static and dynamic stimulation, and is capable of classifying saccades, post-saccadic oscillations, fixations, and smooth pursuit events. We validated classification performance and robustness on three public datasets: 1) manually annotated, trial-based gaze trajectories for viewing static images, moving dots, and short video sequences, 2) lab-quality gaze recordings for a feature length movie, and 3) gaze recordings acquired under suboptimal lighting conditions inside the bore of a magnetic resonance imaging (MRI) scanner for the same full-length movie. We found that the proposed algorithm performs on par or better compared to state-of-the-art alternatives for static stimulation. Moreover, it yields eye movement events with biologically plausible characteristics on prolonged dynamic recordings. Lastly, algorithm performance is robust on data acquired under suboptimal conditions that exhibit a temporally varying noise level. These results indicate that the proposed algorithm is a robust tool with improved classification accuracy across a range of use cases. The algorithm is cross-platform compatible, implemented using the Python programming language, and readily available as free and open source software from public sources.

[1]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[2]  Vinoo Alluri,et al.  Capturing the musical brain with Lasso: Dynamic decoding of musical features from fMRI data , 2014, NeuroImage.

[3]  Tapio Takala,et al.  Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments , 2012, Front. Hum. Neurosci..

[4]  Nergiz Ercil Cagiltay,et al.  Evaluation of Ten Open-Source Eye-Movement Classification Algorithms in Simulated Surgical Scenarios , 2019, IEEE Access.

[5]  Eleanor A. Maguire,et al.  Studying the freely-behaving brain with fMRI , 2012, NeuroImage.

[6]  Wes McKinney,et al.  Data Structures for Statistical Computing in Python , 2010, SciPy.

[7]  P. Jaccard,et al.  Etude comparative de la distribution florale dans une portion des Alpes et des Jura , 1901 .

[8]  Marcus Nyström,et al.  The pupil is faster than the corneal reflection (CR): Are video based pupil-CR eye trackers suitable for studying detailed dynamics of eye movements? , 2016, Vision Research.

[9]  M. Rucci,et al.  Precision of sustained fixation in trained and untrained observers. , 2012, Journal of vision.

[10]  Ignace T. C. Hooge,et al.  Is the eye-movement field confused about fixations and saccades? A survey among 124 researchers , 2018, Royal Society Open Science.

[11]  Oleg V Komogortsev,et al.  Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades , 2013, Behavior research methods.

[12]  Dave M. Stampe,et al.  Heuristic filtering and reliable calibration methods for video-based pupil-tracking systems , 1993 .

[13]  P. Gordon,et al.  Similarity-based interference during language comprehension: Evidence from eye tracking during reading. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[14]  Randolph Blake,et al.  Pupil size dynamics during fixation impact the accuracy and precision of video-based gaze estimation , 2016, Vision Research.

[15]  Diederick C. Niehorster,et al.  Noise-robust fixation detection in eye movement data: Identification by two-means clustering (I2MC) , 2016, Behavior Research Methods.

[16]  Catherine Perrodin,et al.  Are We Ready for Real-world Neuroscience? , 2019, Journal of Cognitive Neuroscience.

[17]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[18]  Deborah E. Hannula,et al.  Worth a Glance: Using Eye Movements to Investigate the Cognitive Neuroscience of Memory , 2010, Front. Hum. Neurosci..

[19]  E. Irving,et al.  Vertical Eye Position Control in Darkness: Orbital Position and Body Orientation Interact to Modulate Drift Velocity , 1997, Vision Research.

[20]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[21]  Lee Friedman,et al.  A novel evaluation of two related and two independent algorithms for eye movement classification during reading , 2018, Behavior Research Methods.

[22]  Ioannis Agtzidis,et al.  1D CNN with BLSTM for automated classification of fixations, saccades, and smooth pursuits , 2018, Behavior Research Methods.

[23]  Alexander C. Schütz,et al.  Eye movements and perception: a selective review. , 2011, Journal of vision.

[24]  Marcus Nyström,et al.  An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data , 2010, Behavior research methods.

[25]  H. Laufs,et al.  Decoding Wakefulness Levels from Typical fMRI Resting-State Data Reveals Reliable Drifts between Wakefulness and Sleep , 2014, Neuron.

[26]  Timothy J. Andrews,et al.  Dynamic stimuli demonstrate a categorical representation of facial expression in the amygdala , 2014, Neuropsychologia.

[27]  Marcus Nyström,et al.  One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms , 2016, Behavior Research Methods.

[28]  Marcus Nyström,et al.  Detection of Saccades and Postsaccadic Oscillations in the Presence of Smooth Pursuit , 2013, IEEE Transactions on Biomedical Engineering.

[29]  L. Stark,et al.  The main sequence, a tool for studying human eye movements , 1975 .

[30]  Scott P. Johnson,et al.  Gazepath: An eye-tracking analysis tool that accounts for individual differences and data quality , 2017, Behavior Research Methods.

[31]  R. Gellman,et al.  Human smooth pursuit: stimulus-dependent responses. , 1987, Journal of neurophysiology.

[32]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[33]  Marcus Nyström,et al.  Detection of fixations and smooth pursuit movements in high-speed eye-tracking data , 2015, Biomed. Signal Process. Control..

[34]  Ingrid Heynderickx,et al.  Visual Attention in Objective Image Quality Assessment: Based on Eye-Tracking Data , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Oleg V. Komogortsev,et al.  Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors , 2010, IEEE Transactions on Biomedical Engineering.

[36]  Skipper Seabold,et al.  Statsmodels: Econometric and Statistical Modeling with Python , 2010, SciPy.

[37]  Marcus Nyström,et al.  Is human classification by experienced untrained observers a gold standard in fixation detection? , 2017, Behavior research methods.

[38]  Kenneth Holmqvist,et al.  gazeNet: End-to-end eye-movement event detection with deep neural networks , 2018, Behavior Research Methods.

[39]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[40]  Cristian Sminchisescu,et al.  Dynamic Eye Movement Datasets and Learnt Saliency Models for Visual Action Recognition , 2012, ECCV.

[41]  Satrajit S. Ghosh,et al.  The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments , 2016, Scientific Data.

[42]  Michael Hanke,et al.  A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation , 2016, Scientific Data.