Robust cognitive load detection from wrist-band sensors

Abstract In recent years, the detection of cognitive load has received a lot of attention. Understanding the circumstances in which cognitive load occurs and reliably predicting such occurrences, offers the potential for considerable advances in the field of Human-Computer Interaction (HCI). Numerous HCI applications, ranging from medical and health-related solutions to (smart) automotive environments, would directly benefit from the reliable detection of cognitive load. However, this task still remains highly challenging. We present a machine learning (ML) approach based on ensemble learning for robust cognitive load classification. The features used by the proposed solution are generated from the interpretation of physiological measurements (e.g., heart rate, r-r interval, skin temperature, and skin response) from a wearable device. Hence, our approach consists of two steps: (1) transforming the original data into discriminative features and (2) training an ensemble model to accurately and robustly predict cognitive load. The empirical results confirm that our method has a superior performance compared to various state-of-the-art baselines on the original and transformed data. Moreover, in the open-data CogLoad@UbiComp 2020 Competition, the proposed approach achieved the best results among 17 competing approaches and outperformed all participating competitors by a considerable margin.

[1]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[2]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[3]  Florian Alt,et al.  Your Eyes Tell: Leveraging Smooth Pursuit for Assessing Cognitive Workload , 2018, CHI.

[4]  Thomas C. Kübler,et al.  Stress-indicators and exploratory gaze for the analysis of hazard perception in patients with visual field loss. , 2014 .

[5]  Jan Ole Johanssen,et al.  Employing Consumer Wearables to Detect Office Workers' Cognitive Load for Interruption Management , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[6]  Sandra P Marshall,et al.  Identifying cognitive state from eye metrics. , 2007, Aviation, space, and environmental medicine.

[7]  KostakosVassilis,et al.  Assessing Cognitive Performance Using Physiological and Facial Features , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[8]  B. Bridgeman,et al.  Microsaccades and Exploratory Saccades in a Naturalistic Environment , 2011 .

[9]  Peter Bloomfield,et al.  Fourier Analysis of Time Series: An Introduction , 1977 .

[11]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[12]  Krzysztof Krejtz,et al.  Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze , 2018, PloS one.

[13]  J. McCarley,et al.  Executive working memory load does not compromise perceptual processing during visual search: Evidence from additive factors analysis , 2010, Attention, perception & psychophysics.

[14]  Gerhard Tröster,et al.  Discriminating Stress From Cognitive Load Using a Wearable EDA Device , 2010, IEEE Transactions on Information Technology in Biomedicine.

[15]  Gjergji Kasneci,et al.  CancelOut: A Layer for Feature Selection in Deep Neural Networks , 2019, ICANN.

[16]  Gjergji Kasneci,et al.  Aggregating physiological and eye tracking signals to predict perception in the absence of ground truth , 2017, Comput. Hum. Behav..

[17]  Yang Wang,et al.  Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks , 2012, OZCHI.

[18]  Enkelejda Kasneci,et al.  Predicting Cognitive Load in an Emergency Simulation Based on Behavioral and Physiological Measures , 2019, ICMI.

[19]  M. W Gardner,et al.  Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences , 1998 .

[20]  T. Gog,et al.  Effects of prior knowledge and concept-map structure on disorientation, cognitive load, and learning , 2009 .

[21]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[22]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[23]  G F Wilson,et al.  Evoked potential, cardiac, blink, and respiration measures of pilot workload in air-to-ground missions. , 1994, Aviation, space, and environmental medicine.

[24]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[25]  Bryan Reimer,et al.  A Comparison of Heart Rate and Heart Rate Variability Indices in Distinguishing Single-Task Driving and Driving Under Secondary Cognitive Workload , 2017 .

[26]  J. Friedman Stochastic gradient boosting , 2002 .

[27]  Erik D. Reichle,et al.  Lexical and Post-Lexical Complexity Effects on Eye Movements in Reading. , 2011, Journal of eye movement research.

[28]  Geraint Rees,et al.  Two distinct neural effects of blinking on human visual processing , 2005, NeuroImage.

[29]  J. Beatty,et al.  The pupillary system. , 2000 .

[30]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[31]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[32]  M. Gams,et al.  Datasets for Cognitive Load Inference Using Wearable Sensors and Psychological Traits , 2020, Applied Sciences.

[33]  F. Vetere,et al.  Cognitive Heat , 2017 .

[34]  Jani Mäntyjärvi,et al.  Ultra-Short Window Length and Feature Importance Analysis for Cognitive Load Detection from Wearable Sensors , 2021, Electronics.

[35]  Michela Terenzi,et al.  A Random Glance at the Flight Deck: Pilots' Scanning Strategies and the Real-Time Assessment of Mental Workload , 2007 .

[36]  Michel Verleysen,et al.  The Curse of Dimensionality in Data Mining and Time Series Prediction , 2005, IWANN.

[37]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[38]  Pradipta Biswas,et al.  Cognitive load estimation using ocular parameters in automotive , 2020 .

[39]  Lior Rokach,et al.  Ensemble-based classifiers , 2010, Artificial Intelligence Review.

[40]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[41]  Jason J. S. Barton,et al.  The effects of enhanced attention and working memory on smooth pursuit eye movement , 2018, Experimental Brain Research.

[42]  U. Lindenberger,et al.  A task is a task is a task: putting complex span, n-back, and other working memory indicators in psychometric context , 2014, Front. Psychol..

[43]  Qiuzhen Wang,et al.  An eye-tracking study of website complexity from cognitive load perspective , 2014, Decis. Support Syst..

[44]  Pradipta Biswas,et al.  Estimating Pilots’ Cognitive Load From Ocular Parameters Through Simulation and In-Flight Studies , 2019, Journal of eye movement research.

[45]  Hong-Jin Sun,et al.  Modulation of microsaccade rate by task difficulty revealed through between- and within-trial comparisons. , 2015, Journal of vision.

[46]  John Sweller,et al.  Cognitive Load Theory , 2020, Encyclopedia of Education and Information Technologies.

[47]  Veljko Pejovic,et al.  My Watch Says I'm Busy: Inferring Cognitive Load with Low-Cost Wearables , 2018, UbiComp/ISWC Adjunct.

[48]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[49]  Tie-Yan Liu,et al.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS.

[50]  Ada Wai-Chee Fu,et al.  Efficient time series matching by wavelets , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[51]  Aslak Grinsted,et al.  Nonlinear Processes in Geophysics Application of the Cross Wavelet Transform and Wavelet Coherence to Geophysical Time Series , 2022 .

[52]  Andrea Salfinger Deep learning for cognitive load monitoring: a comparative evaluation , 2020, UbiComp/ISWC Adjunct.

[53]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[54]  George Karabatis,et al.  Discrete wavelet transform-based time series analysis and mining , 2011, CSUR.

[55]  Hiroshi Ishiguro,et al.  A Non-parametric Approach to the Overall Estimate of Cognitive Load Using NIRS Time Series , 2017, Front. Hum. Neurosci..

[56]  Martine De Cock,et al.  Cognitive load detection from wrist-band sensors , 2020, UbiComp/ISWC Adjunct.

[57]  Nicolas Pinto,et al.  SkData: Data Sets and Algorithm Evaluation Protocols in Python , 2013 .

[58]  Daniel McDuff,et al.  Remote measurement of cognitive stress via heart rate variability , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[59]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[60]  Weidong Huang,et al.  Towards Preventative Healthcare: A Review of Wearable and Mobile Applications , 2018, ICIMTH.

[61]  Kyosuke Fukuda,et al.  Cognition, blinks, eye-movements, and pupillary movements during performance of a running memory task. , 2005, Aviation, space, and environmental medicine.

[62]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[63]  Andrew L. Kun,et al.  On the feasibility of using pupil diameter to estimate cognitive load changes for in-vehicle spoken dialogues , 2013, INTERSPEECH.

[64]  Suku Nair,et al.  Work-in-Progress, PupilWare-M: Cognitive Load Estimation Using Unmodified Smartphone Cameras , 2015, 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems.

[65]  Siyuan Chen,et al.  Eye activity as a measure of human mental effort in HCI , 2011, IUI '11.

[66]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[67]  T. Gog,et al.  Uncovering Expertise-Related Differences in Troubleshooting Performance: Combining Eye Movement and Concurrent Verbal Protocol Data , 2005 .

[68]  Eduardo Salas,et al.  Cardiac Measures of Cognitive Workload: A Meta-Analysis , 2019, Hum. Factors.

[69]  Siyuan Chen,et al.  Using Task-Induced Pupil Diameter and Blink Rate to Infer Cognitive Load , 2014, Hum. Comput. Interact..

[70]  Andrew Olney,et al.  Put your thinking cap on: detecting cognitive load using EEG during learning , 2017, LAK.

[71]  Yang Wang,et al.  Detecting Users’ Cognitive Load by Galvanic Skin Response with Affective Interference , 2017, ACM Trans. Interact. Intell. Syst..

[72]  Marc Pomplun,et al.  Comparative Search Reveals the Tradeoff between Eye Movements and Working Memory Use in Visual Tasks , 2003 .

[73]  Thomas Hagen,et al.  Task context load induces reactive cognitive control: An fMRI study on cortical and brain stem activity , 2019, Cognitive, Affective, & Behavioral Neuroscience.

[74]  Scott Makeig,et al.  Eye Activity Correlates of Workload during a Visuospatial Memory Task , 2001, Hum. Factors.

[75]  Régis Lobjois,et al.  The effects of driving environment complexity and dual tasking on drivers’ mental workload and eye blink behavior , 2016 .

[76]  Philip H. W. Leong,et al.  Grammar-Based Feature Generation for Time-Series Prediction , 2015 .

[77]  Albrecht Schmidt,et al.  A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions , 2016, CHI.

[78]  Martin Raubal,et al.  The Index of Pupillary Activity: Measuring Cognitive Load vis-à-vis Task Difficulty with Pupil Oscillation , 2018, CHI.

[79]  Peter Gerjets,et al.  Cross-subject workload classification using pupil-related measures , 2018, ETRA.

[80]  Junqi Guo,et al.  A data-driven framework for learners' cognitive load detection using ECG-PPG physiological feature fusion and XGBoost classification , 2018, IIKI.

[81]  Matthew Garratt,et al.  Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review , 2019, IEEE Transactions on Cybernetics.

[82]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[83]  Tak-Chung Fu,et al.  A review on time series data mining , 2011, Eng. Appl. Artif. Intell..

[84]  Alex Fridman,et al.  Cognitive Load Estimation in the Wild , 2018, CHI.

[85]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[86]  Francisco M. Costela,et al.  Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes , 2014, The European journal of neuroscience.