Your Eyes Tell: Leveraging Smooth Pursuit for Assessing Cognitive Workload

A common objective for context-aware computing systems is to predict how user interfaces impact user performance regarding their cognitive capabilities. Existing approaches such as questionnaires or pupil dilation measurements either only allow for subjective assessments or are susceptible to environmental influences and user physiology. We address these challenges by exploiting the fact that cognitive workload influences smooth pursuit eye movements. We compared three trajectories and two speeds under different levels of cognitive workload within a user study (N=20). We found higher deviations of gaze points during smooth pursuit eye movements for specific trajectory types at higher cognitive workload levels. Using an SVM classifier, we predict cognitive workload through smooth pursuit with an accuracy of 99.5% for distinguishing between low and high workload as well as an accuracy of 88.1% for estimating workload between three levels of difficulty. We discuss implications and present use cases of how cognition-aware systems benefit from inferring cognitive workload in real-time by smooth pursuit eye movements.

[1]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[2]  Janko Drnovsek,et al.  Non-contact heart rate and heart rate variability measurements: A review , 2014, Biomed. Signal Process. Control..

[3]  G. Barnes,et al.  Cognitive processes involved in smooth pursuit eye movements , 2008, Brain and Cognition.

[4]  Francisco M. Costela,et al.  Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes , 2014, The European journal of neuroscience.

[5]  Florian Alt,et al.  TextPursuits: using text for pursuits-based interaction and calibration on public displays , 2016, UbiComp.

[6]  Pat Hanrahan,et al.  Measuring the task-evoked pupillary response with a remote eye tracker , 2008, ETRA.

[7]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[8]  Linden J. Ball,et al.  Eye tracking in HCI and usability research. , 2006 .

[9]  Yanxia Zhang,et al.  SideWays: a gaze interface for spontaneous interaction with situated displays , 2013, CHI.

[10]  Elisabeth André,et al.  Writing with Your Eye: A Dwell Time Free Writing System Adapted to the Nature of Human Eye Gaze , 2008, PIT.

[11]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[12]  F. Vetere,et al.  Cognitive Heat , 2017 .

[13]  R. Leigh,et al.  The neurology of eye movements , 1984 .

[14]  D. Sparks The brainstem control of saccadic eye movements , 2002, Nature Reviews Neuroscience.

[15]  Gerhard Rigoll,et al.  GazeEverywhere: Enabling Gaze-only User Interaction on an Unmodified Desktop PC in Everyday Scenarios , 2017, CHI.

[16]  Worthy N. Martin,et al.  Human-computer interaction using eye-gaze input , 1989, IEEE Trans. Syst. Man Cybern..

[17]  Hans-Werner Gellersen,et al.  Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements , 2015, UIST.

[18]  Michelle N. Lumicao,et al.  EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. , 2007, Aviation, space, and environmental medicine.

[19]  David Fitzpatrick,et al.  Types of Eye Movements and Their Functions , 2001 .

[20]  Andrew R. A. Conway,et al.  Working memory, attention control, and the N-back task: a question of construct validity. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[21]  Alexander De Luca,et al.  Evaluation of eye-gaze interaction methods for security enhanced PIN-entry , 2007, OZCHI '07.

[22]  R. Engle,et al.  Is working memory capacity task dependent , 1989 .

[23]  Päivi Majaranta,et al.  Twenty years of eye typing: systems and design issues , 2002, ETRA.

[24]  Robert J. K. Jacob,et al.  Evaluation of eye gaze interaction , 2000, CHI.

[25]  Daniel Vogel,et al.  Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users , 2004, UIST '04.

[26]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[27]  Jan-Louis Kruger,et al.  Measuring the impact of subtitles on cognitive load: eye tracking and dynamic audiovisual texts , 2013, ETSA '13.

[28]  Stefan Mattes,et al.  Surrogate Distraction Measurement Techniques: The Lane Change Test , 2009 .

[29]  M. D’Esposito Working memory. , 2008, Handbook of clinical neurology.

[30]  Antti Oulasvirta,et al.  It's Mine, Don't Touch!: interactions at a large multi-touch display in a city centre , 2008, CHI.

[31]  Thomas Hofmann,et al.  Support vector machine learning for interdependent and structured output spaces , 2004, ICML.

[32]  Robert S. Allison,et al.  Thermal Imaging as a Way to Classify Cognitive Workload , 2010, 2010 Canadian Conference on Computer and Robot Vision.

[33]  Bo Dong,et al.  Exploiting Eye Tracking for Smartphone Authentication , 2015, ACNS.

[34]  Ulf Ahlstrom,et al.  Using eye movement activity as a correlate of cognitive workload , 2006 .

[35]  Andreas Bulling,et al.  Cognition-Aware Computing , 2014, IEEE Pervasive Computing.

[36]  N. Mai,et al.  Selective disturbance of movement vision after bilateral brain damage. , 1983, Brain : a journal of neurology.

[37]  Ulrike Pfeil,et al.  Measuring Cognitive Load using Eye Tracking Technology in Visual Computing , 2016, BELIV '16.

[38]  Vsevolod Peysakhovich Study of pupil diameter and eye movements to enhance flight safety. Etude de diamètre pupillaire et de mouvements oculaires pour la sécurité aérienne , 2016 .

[39]  Laura Chamberlain Eye Tracking Methodology; Theory and Practice , 2007 .

[40]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[41]  Oleg V. Komogortsev,et al.  A Comparison of Smooth Pursuit- and Dwell-based Selection at Multiple Levels of Spatial Accuracy , 2017, CHI Extended Abstracts.

[42]  Kristie Lee Young,et al.  Surrogate Distraction Measurement Techniques: The Lane Change Test , 2008 .

[43]  Tilman Dingler,et al.  Cognition-aware systems as mobile personal assistants , 2016, UbiComp Adjunct.

[44]  Hans-Werner Gellersen,et al.  Pursuits: spontaneous interaction with displays based on smooth pursuit eye movement and moving targets , 2013, UbiComp.

[45]  Hans-Werner Gellersen,et al.  Pursuit calibration: making gaze calibration less tedious and more flexible , 2013, UIST.

[46]  R. Ornstein,et al.  Pupillary Responses During Information Processing Vary with Scholastic Aptitude Test Scores , 2022 .

[47]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[48]  A. Pauzie,et al.  A method to assess the driver mental workload: The driving activity load index (DALI) , 2008 .

[49]  H. Collewijn,et al.  Human smooth and saccadic eye movements during voluntary pursuit of different target motions on different backgrounds. , 1984, The Journal of physiology.

[50]  S. Bahar,et al.  Effect of cognitive load on eye-target synchronization during smooth pursuit eye movement , 2011, Brain Research.

[51]  Takehiko Ohno Features of eye gaze interface for selection tasks , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[52]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[53]  Poika Isokoski,et al.  Now Dasher! Dash away!: longitudinal study of fast text entry by Eye Gaze , 2008, ETRA.

[54]  John D. Lee,et al.  Driver Cognitive Distraction Detection Using Eye Movements , 2008 .

[55]  T. Jung,et al.  Task performance and eye activity: predicting behavior relating to cognitive workload. , 2007, Aviation, space, and environmental medicine.

[56]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[57]  M W Mahowald,et al.  Rapid Eye Movement Sleep Behavior Disorder: A Treatable Parasomnia Affecting Older Adults , 1987 .

[58]  Simone Benedetto,et al.  Driver workload and eye blink duration , 2011 .

[59]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[60]  Albrecht Schmidt,et al.  A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions , 2016, CHI.

[61]  D. Gronwall Paced Auditory Serial-Addition Task: A Measure of Recovery from Concussion , 1977, Perceptual and motor skills.

[62]  Florian Alt,et al.  A field study on spontaneous gaze-based interaction with a public display using pursuits , 2015, UbiComp/ISWC Adjunct.

[63]  Hans-Werner Gellersen,et al.  AmbiGaze: Direct Control of Ambient Devices by Gaze , 2016, Conference on Designing Interactive Systems.

[64]  E. Stuyven,et al.  The effect of cognitive load on saccadic eye movements. , 2000, Acta psychologica.

[65]  Johan Engström,et al.  Sensitivity of eye-movement measures to in-vehicle task difficulty , 2005 .

[66]  Poika Isokoski,et al.  Text input methods for eye trackers using off-screen targets , 2000, ETRA.