Car Driver's Sympathetic Reaction Detection Through Electrodermal Activity and Electrocardiogram Measurements

Objective: in this paper we propose a system to detect a subject's sympathetic reaction, which is related to unexpected or challenging events during a car drive. Methods: we use the Electrocardiogram (ECG) signal and the Skin Potential Response (SPR) signal, which has several advantages with respect to other Electrodermal (EDA) signals. We record one SPR signal for each hand, and use an algorithm that, selecting the smoother signal, is able to remove motion artifacts. We extract statistical features from the ECG and SPR signals in order to classify signal segments and identify the presence or absence of emotional events via a Supervised Learning Algorithm. The experiments were carried out in a company which specializes in driving simulator equipment, using a motorized platform and a driving simulator. Different subjects were tested with this setup, with different challenging events happening on predetermined locations on the track. Results: we obtain an Accuracy as high as 79.10% for signal blocks and as high as 91.27% for events. Conclusion: results demonstrate the good performance of the presented system in detecting sympathetic reactions, and the effectiveness of the motion artifact removal procedure. Significance: our work demonstrates the possibility to classify the emotional state of the driver, using the ECG and EDA signals and a slightly invasive setup. In particular, the proposed use of SPR and of the motion artifact removal procedure are crucial for the effectiveness of the system.

[1]  Pamela Zontone,et al.  Dual channel Electrodermal activity sensor for motion artifact removal in car drivers’ stress detection , 2019, 2019 IEEE Sensors Applications Symposium (SAS).

[2]  Kimihiko Nakano,et al.  Biosignal Analysis to Assess Mental Stress in Automatic Driving of Trucks: Palmar Perspiration and Masseter Electromyography , 2015, Sensors.

[3]  Yorgos Goletsis,et al.  Towards Driver's State Recognition on Real Driving Conditions , 2011 .

[4]  Robert Stone,et al.  Affective Recognition in Dynamic and Interactive Virtual Environments , 2020, IEEE Transactions on Affective Computing.

[5]  Riccardo Bernardini,et al.  Driver’s stress detection using Skin Potential Response signals , 2018, Measurement.

[6]  Backman Al,et al.  Health survey of professional drivers. , 1983 .

[7]  Naeem Ramzan,et al.  DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices , 2018, IEEE Journal of Biomedical and Health Informatics.

[8]  Chandran Saravanan,et al.  Advancements and recent trends in emotion recognition using facial image analysis and machine learning models , 2017, 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT).

[9]  Zhi-Long Chen,et al.  Using EEG to detect drivers' emotion with Bayesian Networks , 2010, 2010 International Conference on Machine Learning and Cybernetics.

[10]  Yair Amichai-Hamburger,et al.  Technology and Psychological Well-being , 2011 .

[11]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[12]  Hung T. Nguyen,et al.  Classification of facial-emotion expression in the application of psychotherapy using Viola-Jones and Edge-Histogram of Oriented Gradient , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[13]  S. Cerutti,et al.  Heart rate variability and respiratory sinus arrhythmia assessment of affective states by bivariate autoregressive spectral analysis , 2010, 2010 Computing in Cardiology.

[14]  Nadine Sarter,et al.  The Effects of Data Density, Display Organization, and Stress on Search Performance: An Eye Tracking Study of Clutter , 2017, IEEE Transactions on Human-Machine Systems.

[15]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[16]  Wanhui Wen,et al.  Recognition of Real-Scene Stress in Examination with Heart Rate Features , 2017, 2017 9th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC).

[17]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[18]  A. Affanni Wearable instrument to measure simultaneously cardiac and electrodermal activities , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[19]  Wan-Young Chung,et al.  Wearable Glove-Type Driver Stress Detection Using a Motion Sensor , 2017, IEEE Transactions on Intelligent Transportation Systems.

[20]  A. Backman,et al.  Health survey of professional drivers. , 1983, Scandinavian journal of work, environment & health.

[21]  Giovanni Chiorboli,et al.  Design and characterization of a real-time, wearable, endosomatic electrodermal system , 2015 .

[22]  Yorgos Goletsis,et al.  Real-Time Driver's Stress Event Detection , 2012, IEEE Transactions on Intelligent Transportation Systems.

[23]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[24]  Jesús Lázaro,et al.  Inclusion of Respiratory Frequency Information in Heart Rate Variability Analysis for Stress Assessment , 2016, IEEE Journal of Biomedical and Health Informatics.

[25]  Laurent Albera,et al.  Emotion Recognition Based on High-Resolution EEG Recordings and Reconstructed Brain Sources , 2020, IEEE Transactions on Affective Computing.

[26]  A. Affanni,et al.  Motion artifact removal in stress sensors used in “driver in motion” simulators , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[27]  Stylianos Asteriadis,et al.  High-performance and lightweight real-time deep face emotion recognition , 2017, 2017 12th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP).

[28]  Hajime Asama,et al.  Estimation of stress during car race with factor analysis , 2015, 2015 International Symposium on Micro-NanoMechatronics and Human Science (MHS).

[29]  Alain Pruski,et al.  Emotion recognition for human-machine communication , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[30]  Yihan Zhao,et al.  Stress level evaluation using BP Neural network based on time-frequency analysis of HRV , 2017, 2017 IEEE International Conference on Mechatronics and Automation (ICMA).

[31]  Dae SeokLee,et al.  Stress Events Detection of Driver by Wearable Glove System , 2017, IEEE Sensors Journal.

[32]  Guangyuan Liu,et al.  Detection of Psychological Stress Using a Hyperspectral Imaging Technique , 2014, IEEE Transactions on Affective Computing.

[33]  Naoyuki Kubota,et al.  Classifying Stress From Heart Rate Variability Using Salivary Biomarkers as Reference , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[34]  E. Scilingo,et al.  Arousal and Valence Recognition of Affective Sounds Based on Electrodermal Activity , 2017, IEEE Sensors Journal.

[35]  Filippo Cavallo,et al.  Evaluation of an Integrated System of Wearable Physiological Sensors for Stress Monitoring in Working Environments by Using Biological Markers , 2018, IEEE Transactions on Biomedical Engineering.

[36]  Caroline Schießl Stress and strain while driving , 2007 .

[37]  Willis J. Tompkins,et al.  A Real-Time QRS Detection Algorithm , 1985, IEEE Transactions on Biomedical Engineering.

[38]  James L. Coyle,et al.  Deep Belief Networks for Electroencephalography: A Review of Recent Contributions and Future Outlooks , 2017, IEEE Journal of Biomedical and Health Informatics.

[39]  Hiroshi Mizoguchi,et al.  Real-time eyelid open/closed state recognition based on HLAC towards driver drowsiness detection , 2014, 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014).

[40]  Mohamed Chetouani,et al.  Multimodal Stress Detection from Multiple Assessments , 2018, IEEE Transactions on Affective Computing.

[41]  Riccardo Bernardini,et al.  Low-Complexity Classification Algorithm to Identify Drivers’ Stress Using Electrodermal Activity (EDA) Measurements , 2018 .

[42]  Enzo Pasquale Scilingo,et al.  How the Autonomic Nervous System and Driving Style Change With Incremental Stressing Conditions During Simulated Driving , 2015, IEEE Transactions on Intelligent Transportation Systems.

[43]  Matthias Weippert,et al.  Fuzzy Evaluation of Heart Rate Signals for Mental Stress Assessment , 2007, IEEE Transactions on Fuzzy Systems.

[44]  David D. Cox,et al.  Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[45]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[46]  H Kalvøy,et al.  Electrodermal responses to discrete stimuli measured by skin conductance, skin potential, and skin susceptance , 2018, Skin research and technology : official journal of International Society for Bioengineering and the Skin (ISBS) [and] International Society for Digital Imaging of Skin (ISDIS) [and] International Society for Skin Imaging.

[47]  Beomjoon Kim,et al.  Wearable Mobile-Based Emotional Response-Monitoring System for Drivers , 2017, IEEE Transactions on Human-Machine Systems.

[48]  Gerhard Tröster,et al.  Discriminating Stress From Cognitive Load Using a Wearable EDA Device , 2010, IEEE Transactions on Information Technology in Biomedicine.

[49]  Jinxi Zhang,et al.  Feasibility study on measurement of a physiological index value with an electrocardiogram tester to evaluate the pavement evenness and driving comfort , 2018 .

[50]  Q. M. Jonathan Wu,et al.  EEG-Based Emotion Recognition Using Hierarchical Network With Subnetwork Nodes , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[51]  Gyogwon Koo,et al.  Wearable Device-Based System to Monitor a Driver’s Stress, Fatigue, and Drowsiness , 2018, IEEE Transactions on Instrumentation and Measurement.

[52]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[53]  Rosalind W. Picard,et al.  Multiple Arousal Theory and Daily-Life Electrodermal Activity Asymmetry , 2016 .

[54]  M. P. S. Chawla,et al.  PCA and ICA processing methods for removal of artifacts and noise in electrocardiograms: A survey and comparison , 2011, Appl. Soft Comput..

[55]  Zhiping Lin,et al.  Investigation on driver stress utilizing ECG signals with on-board navigation systems in use , 2016, 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV).

[56]  Carmen C. Y. Poon,et al.  Unobtrusive and Multimodal Wearable Sensing to Quantify Anxiety , 2016, IEEE Sensors Journal.

[57]  James D. Johnston,et al.  Transform coding of audio signals using perceptual noise criteria , 1988, IEEE J. Sel. Areas Commun..