EEGEyeNet: a Simultaneous Electroencephalography and Eye-tracking Dataset and Benchmark for Eye Movement Prediction

We present a new dataset and benchmark with the goal of advancing research in the intersection of brain activities and eye movements. Our dataset, EEGEyeNet, consists of simultaneous Electroencephalography (EEG) and Eye-tracking (ET) recordings from 356 different subjects collected from three different experimental paradigms. Using this dataset, we also propose a benchmark to evaluate gaze prediction from EEG measurements. The benchmark consists of three tasks with an increasing level of difficulty: left-right, angle-amplitude and absolute position. We run extensive experiments on this benchmark in order to provide solid baselines, both based on classical machine learning models and on large neural networks. We release our complete code and data and provide a simple and easy-to-use interface to evaluate new methods.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Steffen Staab,et al.  A multimodal dataset for authoring and editing multimedia content: The MAMEM project , 2017, Data in brief.

[3]  Nora Hollenstein,et al.  ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation , 2020, LREC.

[4]  Nora Hollenstein,et al.  ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading , 2018, Scientific Data.

[5]  Bao-Liang Lu,et al.  A multimodal approach to estimating vigilance using EEG and forehead EOG , 2016, Journal of neural engineering.

[6]  Wojciech Matusik,et al.  Eye Tracking for Everyone , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Kenneth Kreutz-Delgado,et al.  ICLabel: An automated electroencephalographic independent component classifier, dataset, and website , 2019, NeuroImage.

[9]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[10]  Z.O. Abu-Faraj,et al.  Design and Development of a Low-Cost Eye Tracking System for the Rehabilitation of the Completely Locked-In Patient , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[11]  Alain de Cheveigné,et al.  ZapLine: A simple and effective method to remove power line artifacts , 2019, NeuroImage.

[12]  A. Jacobs,et al.  Coregistration of eye movements and EEG in natural reading: analyses and review. , 2011, Journal of experimental psychology. General.

[13]  N. Langer,et al.  Aging Effects and Test–Retest Reliability of Inhibitory Control for Saccadic Eye Movements , 2020, eNeuro.

[14]  Thomas P O'Connell,et al.  Predicting eye movement patterns from fMRI responses to natural scenes , 2018, Nature Communications.

[15]  Howell O. Istance,et al.  Introducing COGAIN: communication by gaze interaction , 2007, Universal Access in the Information Society.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Yu Fang,et al.  Saliency-based gaze prediction based on head direction , 2015, Vision Research.

[18]  Jorge A. Balazs,et al.  Combining eye tracking, pupil dilation and EEG analysis for predicting web users click intention , 2017, Inf. Fusion.

[19]  Geoffrey I. Webb,et al.  InceptionTime: Finding AlexNet for time series classification , 2019, Data Mining and Knowledge Discovery.

[20]  Kristian Lukander,et al.  A probabilistic real-time algorithm for detecting blinks, saccades, and fixations from EOG data , 2015 .

[21]  Christopher Kennard,et al.  An internationally standardised antisaccade protocol , 2013, Vision Research.

[22]  Stephen M. Gordon,et al.  EEGNet: A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces , 2021 .

[23]  Päivi Majaranta,et al.  Eye Tracking and Eye-Based Human–Computer Interaction , 2014 .

[24]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[25]  Jennifer C. Romano Bergstrom,et al.  Eye tracking in user experience design , 2014 .

[26]  Lucas C Parra,et al.  A resource for assessing information processing in the developing brain using EEG and eye tracking , 2016, Scientific Data.

[27]  Alexandros André Chaaraoui,et al.  A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living , 2012, Expert Syst. Appl..

[28]  Nicolas Langer,et al.  Automagic: Standardized Preprocessing of Big EEG Data , 2018 .

[29]  S. Luck,et al.  The effects of electrode impedance on data quality and statistical significance in ERP recordings. , 2010, Psychophysiology.

[30]  Dominic W. Hughes,et al.  Eye-Tracking Patterns in Schizophrenia , 1973, Science.

[31]  W. Klimesch,et al.  EEG alpha oscillations: The inhibition–timing hypothesis , 2007, Brain Research Reviews.

[32]  Bernd Porr,et al.  Real-time estimation of horizontal gaze angle by saccade integration using in-ear electrooculography , 2018, PloS one.

[33]  Florian Alt,et al.  StARe: Gaze-Assisted Face-to-Face Communication in Augmented Reality , 2020, ETRA Adjunct.

[34]  P. König,et al.  Combining EEG and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data , 2012, Front. Hum. Neurosci..

[35]  Yijun Wang,et al.  Combining Brain-Computer Interface and Eye Tracking for High-Speed Text Entry in Virtual Reality , 2018, IUI.

[36]  Weixing Wang,et al.  Driver Fatigue Detection Based on Eye Tracking , 2006, 2006 6th World Congress on Intelligent Control and Automation.

[37]  D. Wechsler Wechsler Adult Intelligence Scale , 2021, Encyclopedia of Evolutionary Psychological Science.

[38]  Michel Wedel,et al.  Eye tracking for visual marketing , 2008 .

[39]  Edward Awh,et al.  Alpha-Band Oscillations Enable Spatially and Temporally Resolved Tracking of Covert Spatial Attention , 2017, Psychological science.

[40]  D. Wechsler Wechsler Intelligence Scale for Children , 2020, Definitions.

[41]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Maria K. Eckstein,et al.  Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? , 2016, Developmental Cognitive Neuroscience.

[43]  Sven Bölte,et al.  Eye tracking in early autism research , 2013, Journal of Neurodevelopmental Disorders.

[44]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[45]  G. A. Miller,et al.  Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalography. , 2014, Psychophysiology.

[46]  A. A. Fedorova,et al.  EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface , 2016, Front. Neurosci..

[47]  Roger Wattenhofer,et al.  Using Deep Learning to Classify Saccade Direction from Brain Activity , 2021, ETRA Short Papers.

[48]  R. Loewenson,et al.  Eye tracking dysfunction in Alzheimer-type dementia. , 1984, Neurology.

[49]  H. Shibasaki,et al.  Oculomotor abnormalities in Parkinson's disease. , 1979, Archives of neurology.

[50]  Norimichi Tsumura,et al.  Advertisement Effectiveness Estimation Based on Crowdsourced Multimodal Affective Responses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[51]  S. Martinez-Conde,et al.  The impact of microsaccades on vision: towards a unified theory of saccadic function , 2013, Nature Reviews Neuroscience.

[52]  Ting Xu,et al.  Evaluating fMRI-Based Estimation of Eye Gaze during Naturalistic Viewing , 2018, bioRxiv.

[53]  Gina M. Notaro,et al.  Simultaneous EEG, eye-tracking, behavioral, and screen-capture data during online German language learning , 2018, Data in brief.

[54]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.