A framework to estimate cognitive load using physiological data

Cognitive load has been widely studied to help understand human performance. It is desirable to monitor user cognitive load in applications such as automation, robotics, and aerospace to achieve operational safety and to improve user experience. This can allow efficient workload management and can help to avoid or to reduce human error. However, tracking cognitive load in real time with high accuracy remains a challenge. Hence, we propose a framework to detect cognitive load by non-intrusively measuring physiological data from the eyes and heart. We exemplify and evaluate the framework where participants engage in a task that induces different levels of cognitive load. The framework uses a set of classifiers to accurately predict low, medium and high levels of cognitive load. The classifiers achieve high predictive accuracy. In particular, Random Forest and Naive Bayes performed best with accuracies of 91.66% and 85.83% respectively. Furthermore, we found that, while mean pupil diameter change for both right and left eye were the most prominent features, blinking rate also made a moderately important contribution to this highly accurate prediction of low, medium and high cognitive load. The existing results on accuracy considerably outperform prior approaches and demonstrate the applicability of our framework to detect cognitive load.

[1]  R. Hughson,et al.  Heart rate variability at rest and exercise: influence of age, gender, and physical training. , 1996, Canadian journal of applied physiology = Revue canadienne de physiologie appliquee.

[2]  Stefan Frank,et al.  Early effects of word surprisal on pupil size during reading , 2012, CogSci.

[3]  Alex Fridman,et al.  Cognitive Load Estimation in the Wild , 2018, CHI.

[4]  Jiming Liu,et al.  Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range , 2014, BMC Medical Research Methodology.

[5]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[6]  Michael E. Smith,et al.  Monitoring Working Memory Load during Computer-Based Tasks with EEG Pattern Recognition Methods , 1998, Hum. Factors.

[7]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[8]  J. Sweller COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[9]  Fang Chen,et al.  Speech-based cognitive load monitoring system , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[10]  Christophe Ley,et al.  Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median , 2013 .

[11]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[12]  A. K. Blangsted,et al.  The effect of mental stress on heart rate variability and blood pressure during computer work , 2004, European Journal of Applied Physiology.

[13]  Riender Happee,et al.  Automated driving reduces perceived workload, but monitoring causes higher cognitive load than manual driving , 2019, Transportation Research Part F: Traffic Psychology and Behaviour.

[14]  Jodi Forlizzi,et al.  Psycho-physiological measures for assessing cognitive load , 2010, UbiComp.

[15]  Fang Chen,et al.  Using language complexity to measure cognitive load for adaptive interaction design , 2010, IUI '10.

[16]  R. Mayer,et al.  Nine Ways to Reduce Cognitive Load in Multimedia Learning , 2003 .

[17]  Nathaniel P. Grove,et al.  Measuring Load on Working Memory: The Use of Heart Rate as a Means of Measuring Chemistry Students' Cognitive Load , 2014 .

[18]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[19]  Helen F. Hastie,et al.  Towards a Conversational Agent for Remote Robot-Human Teaming , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[20]  Yang Wang,et al.  Robust Multimodal Cognitive Load Measurement , 2016, Human–Computer Interaction Series.

[21]  A. Monster,et al.  Long-term trends in human eye blink rate. , 1978, Biotelemetry and patient monitoring.

[22]  Fang Chen,et al.  Think before you talk: an empirical study of relationship between speech pauses and cognitive load , 2008, OZCHI.

[23]  Charles H Morris,et al.  Pilot mental workload: how well do pilots really perform? , 2006, Ergonomics.

[24]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[25]  Manfred Tscheligi,et al.  Everyday Automation Experience: Non-Expert Users Encountering Ubiquitous Automated Systems , 2019, CHI Extended Abstracts.

[26]  Mariska E. Kret,et al.  Preprocessing pupil size data: Guidelines and code , 2018, Behavior Research Methods.

[27]  B. Ainsworth,et al.  International physical activity questionnaire: 12-country reliability and validity. , 2003, Medicine and science in sports and exercise.

[28]  C. Bonnin-Arias,et al.  Pupillary behavior in relation to wavelength and age , 2014, Front. Hum. Neurosci..

[29]  Julie A. Adams,et al.  A Survey of Workload Assessment Algorithms , 2018, IEEE Transactions on Human-Machine Systems.

[30]  Frank Broz,et al.  Gaze and Attention During an HRI Storytelling Task , 2015, HRI 2015.

[31]  Mohd Fauzi Othman,et al.  INDUSTRY 4.0: A REVIEW ON INDUSTRIAL AUTOMATION AND ROBOTIC , 2016 .

[32]  Jamie Reilly,et al.  The human task-evoked pupillary response function is linear: Implications for baseline response scaling in pupillometry , 2018, Behavior Research Methods.

[33]  Heikki Kyröläinen,et al.  International Physical Activity Questionnaire: Validity against fitness. , 2006, Medicine and science in sports and exercise.

[34]  David A. Robb,et al.  Challenges in Collaborative HRI for Remote Robot Teams , 2019, ArXiv.

[35]  Ayoub Al-Hamadi,et al.  Effects of Video Encoding on Camera-Based Heart Rate Estimation , 2019, IEEE Transactions on Biomedical Engineering.

[36]  P. Chandler,et al.  Cognitive Load While Learning to Use a Computer Program , 1996 .

[37]  Jianlong Zhou,et al.  Galvanic Skin Response-Based Measures , 2016 .

[38]  M. Holland,et al.  Blinking and Mental Load , 1972, Psychological reports.

[39]  Rubin Wang,et al.  Recognition of Mental Workload Levels Under Complex Human–Machine Collaboration by Using Physiological Features and Adaptive Support Vector Machines , 2015, IEEE Transactions on Human-Machine Systems.

[40]  Kenneth W. Bauer,et al.  Improving pilot mental workload classification through feature exploitation and combination: a feasibility study , 2005, Comput. Oper. Res..

[41]  Christian A. Müller,et al.  Assessment of a User's Time Pressure and Cognitive Load on the Basis of Features of Speech , 2011, Resource-Adaptive Cognitive Processes.

[42]  Bo Pang,et al.  A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts , 2004, ACL.

[43]  T. Gog,et al.  Development of an instrument for measuring different types of cognitive load , 2013, Behavior Research Methods.

[44]  Fang Chen,et al.  Analysis of Collaborative Communication for Linguistic Cues of Cognitive Load , 2012, Hum. Factors.

[45]  S. Steinhauer,et al.  Blink before and after you think: blinks occur prior to and following cognitive load indexed by pupillary responses. , 2008, Psychophysiology.

[46]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[47]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[48]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[49]  B. Oken,et al.  Sensitivity to mental effort and test–retest reliability of heart rate variability measures in healthy seniors , 2011, Clinical Neurophysiology.

[50]  Helen F. Hastie,et al.  Symptoms of cognitive load in interactions with a dialogue system , 2018, MCPMD@ICMI.

[51]  Anthony Jameson,et al.  Interpreting symptoms of cognitive load in speech input , 1999 .

[52]  Yang Wang,et al.  Eye-Based Measures , 2016 .