Wearable Technologies for Mental Workload, Stress, and Emotional State Assessment during Working-Like Tasks: A Comparison with Laboratory Technologies

The capability of monitoring user’s performance represents a crucial aspect to improve safety and efficiency of several human-related activities. Human errors are indeed among the major causes of work-related accidents. Assessing human factors (HFs) could prevent these accidents through specific neurophysiological signals’ evaluation but laboratory sensors require highly-specialized operators and imply a certain grade of invasiveness which could negatively interfere with the worker’s activity. On the contrary, consumer wearables are characterized by their ease of use and their comfortability, other than being cheaper compared to laboratory technologies. Therefore, wearable sensors could represent an ideal substitute for laboratory technologies for a real-time assessment of human performances in ecological settings. The present study aimed at assessing the reliability and capability of consumer wearable devices (i.e., Empatica E4 and Muse 2) in discriminating specific mental states compared to laboratory equipment. The electrooculographic (EOG), electrodermal activity (EDA) and photoplethysmographic (PPG) signals were acquired from a group of 17 volunteers who took part to the experimental protocol in which different working scenarios were simulated to induce different levels of mental workload, stress, and emotional state. The results demonstrated that the parameters computed by the consumer wearable and laboratory sensors were positively and significantly correlated and exhibited the same evidences in terms of mental states discrimination.

[1]  Mohammed Abo-Zahhad,et al.  Eye Blinking EOG Signals as Biometrics , 2017 .

[2]  Willis J. Tompkins,et al.  A Real-Time QRS Detection Algorithm , 1985, IEEE Transactions on Biomedical Engineering.

[3]  Susanne Fischer,et al.  Intra-individual psychological and physiological responses to acute laboratory stressors of different intensity , 2015, Psychoneuroendocrinology.

[4]  O. Kondrateva,et al.  Development of the Methods for Assessing the Risk of Damage to Health of the Employees of the Electric Power Industry , 2019, Occupational Safety in Industry.

[5]  Nico Pallamin,et al.  Emotion Recognition Using Physiological Signals: Laboratory vs. Wearable Sensors , 2017, AHFE.

[6]  T. Wall,et al.  On the validity of subjective measures of company performance , 2004 .

[7]  Alejandro de la Torre-Luque,et al.  Acute stress recovery through listening to Melomics relaxing music: A randomized controlled trial , 2017 .

[8]  Jaehun Joo,et al.  Consumer adaptation and infusion of wearable devices for healthcare , 2017, Comput. Hum. Behav..

[9]  Philippe Ravaud,et al.  Patients’ views of wearable devices and AI in healthcare: findings from the ComPaRe e-cohort , 2019, npj Digital Medicine.

[10]  Fabio Babiloni,et al.  How Neurophysiological Measures Can be Used to Enhance the Evaluation of Remote Tower Solutions , 2019, Front. Hum. Neurosci..

[11]  Maurizio Bevilacqua,et al.  Human factor risk management in the process industry: A case study , 2018, Reliab. Eng. Syst. Saf..

[12]  M. Stella Atkins,et al.  Workload assessment of surgeons: correlation between NASA TLX and blinks , 2012, Surgical Endoscopy.

[13]  Conor McGinn,et al.  Using the Geneva Emotion Wheel to Measure Perceived Affect in Human-Robot Interaction , 2020, 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[14]  Fabio Babiloni,et al.  The Dry Revolution: Evaluation of Three Different EEG Dry Electrode Types in Terms of Signal Spectral Features, Mental States Classification and Usability , 2019, Sensors.

[15]  Mehdi Jahangiri,et al.  Human Error Analysis in a Permit to Work System: A Case Study in a Chemical Plant , 2015, Safety and health at work.

[16]  Thanos G. Stavropoulos,et al.  Human Factors and Requirements of People with Mild Cognitive Impairment, their Caregivers and Healthcare Professionals for eHealth Systems with Wearable Trackers , 2020, 2020 IEEE International Conference on Human-Machine Systems (ICHMS).

[17]  Fabio Babiloni,et al.  Mental Workload Monitoring: New Perspectives from Neuroscience , 2019, H-WORKLOAD.

[18]  Fillia Makedon,et al.  Brain-EE: Brain Enjoyment Evaluation using Commercial EEG Headband , 2016, PETRA.

[19]  Robert Richer,et al.  An Overview of the Feasibility of Permanent, Real-Time, Unobtrusive Stress Measurement with Current Wearables , 2019, PervasiveHealth.

[20]  Frederick D. Hansen,et al.  Human Error: A Concept Analysis , 2006 .

[21]  Ryan J. Giuliano,et al.  Self-Compassion and Psycho-Physiological Recovery From Recalled Sport Failure , 2019, Front. Psychol..

[22]  Jaewoo Lim,et al.  Evolution of Wearable Devices with Real-Time Disease Monitoring for Personalized Healthcare , 2019, Nanomaterials.

[23]  Fabio Babiloni,et al.  A new regression-based method for the eye blinks artifacts correction in the EEG signal, without using any EOG channel , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[24]  T. Hastie,et al.  Accuracy in Wrist-Worn, Sensor-Based Measurements of Heart Rate and Energy Expenditure in a Diverse Cohort , 2016, bioRxiv.

[25]  W. Kirchner.  Age differences in short-term retention of rapidly changing information. , 1958, Journal of experimental psychology.

[26]  G. Borghini,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[27]  Michelle N. Lumicao,et al.  EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. , 2007, Aviation, space, and environmental medicine.

[28]  Fabio Babiloni,et al.  Mental workload estimations in unilateral deafened children , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[29]  Shyamal Patel,et al.  A review of wearable sensors and systems with application in rehabilitation , 2012, Journal of NeuroEngineering and Rehabilitation.

[30]  Gerhard Tröster,et al.  Discriminating Stress From Cognitive Load Using a Wearable EDA Device , 2010, IEEE Transactions on Information Technology in Biomedicine.

[31]  Peter A Hancock,et al.  State of science: mental workload in ergonomics , 2015, Ergonomics.

[32]  Vladimir J. Konečni,et al.  Comparative effects of music and recalled life-events on emotional state , 2008 .

[33]  D. Bach.  A head-to-head comparison of SCRalyze and Ledalab, two model-based methods for skin conductance analysis , 2014, Biological Psychology.

[34]  Andy Adler,et al.  Validation of the Empatica E4 wristband , 2016, 2016 IEEE EMBS International Student Conference (ISC).

[35]  Michela Sarlo,et al.  Stressing the accuracy: Wrist-worn wearable sensor validation over different conditions. , 2019, Psychophysiology.

[36]  Fabio Babiloni,et al.  Reliability over time of EEG-based mental workload evaluation during Air Traffic Management (ATM) tasks , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[37]  M. Benedek,et al.  A continuous measure of phasic electrodermal activity , 2010, Journal of Neuroscience Methods.

[38]  J. Russell,et al.  Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. , 1999, Journal of personality and social psychology.

[39]  Fabio Babiloni,et al.  Monitoring performance of professional and occupational operators. , 2020, Handbook of clinical neurology.

[40]  Julia C. Lo,et al.  Measuring Mental Workload With Low-Cost and Wearable Sensors: Insights Into the Accuracy, Obtrusiveness, and Research Usability of Three Instruments , 2017, Journal of cognitive engineering and decision making.

[41]  Fabio Babiloni,et al.  Human Factors and Neurophysiological Metrics in Air Traffic Control: A Critical Review , 2017, IEEE Reviews in Biomedical Engineering.

[42]  Régis Lobjois,et al.  The effects of driving environment complexity and dual tasking on drivers’ mental workload and eye blink behavior , 2016 .

[43]  Fabio Babiloni,et al.  Brain Interaction during Cooperation: Evaluating Local Properties of Multiple-Brain Network , 2017, Brain sciences.

[44]  Tiwana Varrecchia,et al.  Wearable Monitoring Devices for Biomechanical Risk Assessment at Work: Current Status and Future Challenges—A Systematic Review , 2018, International journal of environmental research and public health.

[45]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[46]  H. G. Goovaerts,et al.  A Digital QRS Detector Based on the Principle of Contour Limiting , 1976, IEEE Transactions on Biomedical Engineering.

[47]  Rebecca A. Grier.  How High is High? A Meta-Analysis of NASA-TLX Global Workload Scores , 2015 .

[48]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[49]  J. Bakdash,et al.  Repeated Measures Correlation , 2017, Front. Psychol..

[50]  M. Haghi,et al.  Wearable Devices in Medical Internet of Things: Scientific Research and Commercially Available Devices , 2017, Healthcare informatics research.

[51]  Candace M Burns,et al.  The Impact of an Aging Population in the Workplace , 2018, Workplace health & safety.

[52]  Scott E. Kerick,et al.  BLINKER: Automated Extraction of Ocular Indices from EEG Enabling Large-Scale Analysis , 2017, Frontiers in neuroscience.

[53]  Fabio Babiloni,et al.  Stress Assessment by Combining Neurophysiological Signals and Radio Communications of Air Traffic Controllers , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[54]  Fabio Babiloni,et al.  How to Measure Cerebral Correlates of Emotions in Marketing Relevant Tasks , 2014, Cognitive Computation.

[55]  Fabio Babiloni,et al.  EEG-Based Cognitive Control Behaviour Assessment: an Ecological study with Professional Air Traffic Controllers , 2017, Scientific Reports.

[56]  Fabio Babiloni,et al.  A multimodal and signals fusion approach for assessing the impact of stressful events on Air Traffic Controllers , 2020, Scientific Reports.

[57]  C. Debout.  [Concept analysis]. , 2015, Soins; la revue de reference infirmiere.

[58]  E. Dooley,et al.  Estimating Accuracy at Exercise Intensities: A Comparative Study of Self-Monitoring Heart Rate and Physical Activity Wearable Devices , 2017, JMIR mHealth and uHealth.

[59]  Chad C. Williams,et al.  Choosing MUSE: Validation of a Low-Cost, Portable EEG System for ERP Research , 2017, Front. Neurosci..

[60]  Laura Astolfi,et al.  Frontal EEG theta changes assess the training improvements of novices in flight simulation tasks , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).