Using Psychophysiological Sensors to Assess Mental Workload During Web Browsing

Knowledge of the mental workload induced by a Web page is essential for improving users’ browsing experience. However, continuously assessing the mental workload during a browsing task is challenging. To address this issue, this paper leverages the correlation between stimuli and physiological responses, which are measured with high-frequency, non-invasive psychophysiological sensors during very short span windows. An experiment was conducted to identify levels of mental workload through the analysis of pupil dilation measured by an eye-tracking sensor. In addition, a method was developed to classify mental workload by appropriately combining different signals (electrodermal activity (EDA), electrocardiogram, photoplethysmo-graphy (PPG), electroencephalogram (EEG), temperature and pupil dilation) obtained with non-invasive psychophysiological sensors. The results show that the Web browsing task involves four levels of mental workload. Also, by combining all the sensors, the efficiency of the classification reaches 93.7%.

[1]  Isabelle Guyon,et al.  An Introduction to Feature Extraction , 2006, Feature Extraction.

[2]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[3]  D. Kahneman.  Pupillary Responses in a Pitch-Discrimination Task , 1967 .

[4]  Fabio Rusconi,et al.  The Importance of Human Mental Workload in Web Design , 2012, WEBIST.

[5]  Bryan Reimer,et al.  Sensitivity of Physiological Measures for Detecting Systematic Variations in Cognitive Demand From a Working Memory Task , 2012, Hum. Factors.

[6]  Jodi Forlizzi,et al.  Psycho-physiological measures for assessing cognitive load , 2010, UbiComp.

[7]  Nina Zumel,et al.  Practical Data Science with R , 2014 .

[8]  H. Parsons.  What Happened at Hawthorne? , 1974, Science.

[9]  Jacek Gwizdka,et al.  Using Wireless EEG Signals to Assess Memory Workload in the $n$-Back Task , 2016, IEEE Transactions on Human-Machine Systems.

[10]  Nigel H. Lovell,et al.  Beyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload , 2015, IEEE Transactions on Autonomous Mental Development.

[11]  Minoru Nakayama,et al.  The act of task difficulty and eye-movement frequency for the 'Oculo-motor indices' , 2002, ETRA.

[12]  Enzo Pasquale Scilingo,et al.  Characterizing psychological dimensions in non-pathological subjects through autonomic nervous system dynamics , 2015, Front. Comput. Neurosci..

[13]  William P. Batsford.  The Theoretical Basis of Electrocardiology , 1978, The Yale Journal of Biology and Medicine.

[14]  Rana Fayyaz Ahmad,et al.  Feature extraction and classification for EEG signals using wavelet transform and machine learning techniques , 2015, Australasian Physical & Engineering Sciences in Medicine.

[15]  Brian P. Bailey,et al.  On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state , 2006, Comput. Hum. Behav..

[16]  Alessandro Tognetti,et al.  Inference of human affective states from psychophysiological measurements extracted under ecologically valid conditions , 2014, Front. Neurosci..

[17]  Ifeyinwa E. Achumba,et al.  Monitoring: Taxonomy of Issues, Techniques, Applications, Challenges and Limitations , 2013 .

[18]  P. Tibbetts.  :Cognitive Neuroscience: The Biology of the Mind , 2009 .

[19]  Daniel J. Barber,et al.  The Psychometrics of Mental Workload , 2015, Hum. Factors.

[20]  Terry C. Lansdown,et al.  The mind's eye: cognitive and applied aspects of eye movement research , 2005 .

[21]  Li Zhiwei,et al.  Classification of Mental Task EEG Signals Using Wavelet Packet Entropy and SVM , 2007, 2007 8th International Conference on Electronic Measurement and Instruments.

[22]  Fumio Mizoguchi,et al.  Classifying Cognitive Load and Driving Situation with Machine Learning , 2014 .

[23]  Begoña García Zapirain,et al.  A Stress Sensor Based on Galvanic Skin Response (GSR) Controlled by ZigBee , 2012, Sensors.

[24]  Almir Vieira Dibai Filho.  Avanços da imagem infravermelha na disfunção temporomandibular , 2012 .

[25]  Chong Jin Ong,et al.  A Feature Selection Method for Multilevel Mental Fatigue EEG Classification , 2007, IEEE Transactions on Biomedical Engineering.

[26]  Paul Tibbetts,et al.  :Cognitive Neuroscience: The Biology of the Mind , 2009 .

[27]  J. Andreassi.  Psychophysiology: Human Behavior and Physiological Response , 1980 .

[28]  Meredith Ringel Morris,et al.  What do you see when you're surfing?: using eye tracking to predict salient regions of web pages , 2009, CHI.

[29]  Manfred Velden,et al.  The pupillary response to mental overload , 1977 .

[30]  D. Norman,et al.  Psychological Issues in Support of Multiple Activities , 1986 .

[31]  In-Young Ko,et al.  Cognitive resource-aware unobtrusive service provisioning in ambient intelligence environments , 2015, J. Ambient Intell. Smart Environ..

[32]  Yang Wang,et al.  Measurable Decision Making with GSR and Pupillary Analysis for Intelligent User Interface , 2015, ACM Trans. Comput. Hum. Interact..

[33]  Pavlo D. Antonenko,et al.  Using Electroencephalography to Measure Cognitive Load , 2010 .

[34]  G. Thierry,et al.  ERP-pupil size correlations reveal how bilingualism enhances cognitive flexibility , 2013, Cortex.

[35]  Amy Loutfi,et al.  Data Mining for Wearable Sensors in Health Monitoring Systems: A Review of Recent Trends and Challenges , 2013, Sensors.

[36]  James C. Christensen,et al.  Classifying Workload with Eye Movements in a Complex Task , 2012 .

[37]  Christopher A. Monk,et al.  The Attentional Costs of Interrupting Task Performance at Various Stages , 2002 .

[38]  Enzo Pasquale Scilingo,et al.  Emotions and Mood States: Modeling, Elicitation, and Recognition , 2016 .

[39]  J. Fawcett.  The Handbook of Attention , 2015 .

[40]  Yoshihisa Fujiwara,et al.  Using facial skin temperature to objectively evaluate sensations , 1997 .

[41]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[42]  Stéphane Bonnet,et al.  Mental fatigue and working memory load estimation: Interaction and implications for EEG-based passive BCI , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[43]  Virpi Roto,et al.  Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI , 2005, CHI.

[44]  Brian P. Bailey,et al.  Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management , 2008, TCHI.

[45]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[46]  Fernando Seoane,et al.  Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones , 2015, Sensors.

[47]  Enzo Pasquale Scilingo,et al.  Advances in Electrodermal Activity Processing with Applications for Mental Health , 2016, Springer International Publishing.

[48]  T. Caliński,et al.  A dendrite method for cluster analysis , 1974 .

[49]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[50]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[51]  A Latorella Kara,et al.  Effects of Modality on Interrupted Flight Deck Performance: Implications for Data Link , 1998 .

[52]  Susan T. Dumais,et al.  Individual differences in gaze patterns for web search , 2010, IIiX.

[53]  P. Kidwell,et al.  The trouble with computers: Usefulness, usability and productivity , 1996, IEEE Annals of the History of Computing.

[54]  Xun Chen,et al.  Classification of EEG Signals Using a Multiple Kernel Learning Support Vector Machine , 2014, Sensors.

[55]  Ulrich Amsel,et al.  Comprehensive Electrocardiology Theory And Practice In Health And Disease , 2016 .

[56]  W. Levelt,et al.  Pupillary dilation as a measure of attention: a quantitative system analysis , 1993 .

[57]  Robert D. Ward,et al.  Physiological responses to different WEB page designs , 2003, Int. J. Hum. Comput. Stud..

[58]  Joseph H. Goldberg,et al.  Eye Tracking in Usability Evaluation , 2003 .

[59]  Jacek Gwizdka,et al.  Distribution of cognitive load in Web search , 2010, J. Assoc. Inf. Sci. Technol..

[60]  Susan T. Dumais,et al.  The good, the bad, and the random: an eye-tracking study of ad quality in web search , 2010, SIGIR.

[61]  A. Gelenberg,et al.  If not now, when? , 1992, The Journal of clinical psychiatry.

[62]  Joel S. Warm,et al.  Task Engagement and Attentional Resources , 2017, Hum. Factors.

[63]  R. Pea.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1987 .

[64]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[65]  Juan D. Velásquez,et al.  Combining eye tracking and pupillary dilation analysis to identify Website Key Objects , 2015, Neurocomputing.

[66]  Zhilin Zhang,et al.  A Robust Random Forest-Based Approach for Heart Rate Monitoring Using Photoplethysmography Signal Contaminated by Intense Motion Artifacts , 2017, Sensors.

[67]  Yang Wang,et al.  Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks , 2012, OZCHI.

[68]  Daniel Gopher,et al.  On the Economy of the Human Processing System: A Model of Multiple Capacity. , 1977 .

[69]  F. Paas,et al.  Memory load and the cognitive pupillary response in aging. , 2004, Psychophysiology.

[70]  Elizabeth F. Churchill,et al.  Mouse tracking: measuring and predicting users' experience of web-based content , 2012, CHI.

[71]  S. Steinhauer,et al.  Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[72]  Fang Chen,et al.  Pupillary Response Based Cognitive Workload Measurement under Luminance Changes , 2011, INTERACT.

[73]  Gang Li,et al.  Smartwatch-Based Wearable EEG System for Driver Drowsiness Detection , 2015, IEEE Sensors Journal.

[74]  Austin J. Brockmeier,et al.  Learning Recurrent Waveforms Within EEGs , 2016, IEEE Transactions on Biomedical Engineering.

[75]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[76]  Joseph S. Valacich,et al.  The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective , 1999 .

[77]  F. Zijlstra,et al.  Temporal factors in mental work: Effects of interrupted activities , 1999 .

[78]  Stephen J. Guastello,et al.  Individual differences in the experience of cognitive workload , 2015 .

[79]  Eric Kandel,et al.  Biology of the mind. , 2006, Newsweek.

[80]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[81]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[82]  Aamir Saeed Malik,et al.  Classification of cognitive and resting states of the brain using EEG features , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[83]  Michael J. Albers.  Tapping as a measure of cognitive load and website usability , 2011, SIGDOC '11.

[84]  Martha E. Crosby,et al.  Assessing Cognitive Load with Physiological Sensors , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[85]  W. Klimesch.  EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis , 1999, Brain Research Reviews.

[86]  Antti Oulasvirta,et al.  Predicting time-sharing in mobile interaction , 2007, User Modeling and User-Adapted Interaction.

[87]  Peter A Hancock,et al.  State of science: mental workload in ergonomics , 2015, Ergonomics.

[88]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[89]  Andrew Begel,et al.  Using psycho-physiological measures to assess task difficulty in software development , 2014, ICSE.

[90]  Rabab Kreidieh Ward,et al.  An Energy Efficient Compressed Sensing Framework for the Compression of Electroencephalogram Signals , 2014, Sensors.