A survey of machine learning techniques in physiology based mental stress detection systems

Abstract Various automated/semi-automated medical diagnosis systems based on human physiology have been gaining enormous popularity and importance in recent years. Physiological features exhibit several unique characteristics that contribute to reliability, accuracy and robustness of systems. There has also been significant research focusing on detection of conventional positive and negative emotions after presenting laboratory-based stimuli to participants. This paper presents a comprehensive survey on the following facets of mental stress detection systems: physiological data collection, role of machine learning in Emotion Detection systems and Stress Detection systems, various evaluation measures, challenges and applications. An overview of popular feature selection methods is also presented. An important contribution is the exploration of links between biological features of humans with their emotions and mental stress. The numerous research gaps in this field are highlighted which shall pave path for future research.

[1]  Singapore,et al.  Emotion Classification from EEG Signals Using Time-Frequency-DWT Features and ANN , 2017 .

[2]  Masashi Kiguchi,et al.  Stress Assessment Based on Decision Fusion of EEG and fNIRS Signals , 2017, IEEE Access.

[3]  Xianxiang Chen,et al.  Respiration-based emotion recognition with deep learning , 2017, Comput. Ind..

[4]  Srinivas Sethi,et al.  Functional Analysis of Mental Stress Based on Physiological Data of GSR Sensor , 2015 .

[5]  L. H. Viet,et al.  Emotion Detection in the Loop from Brain Signals and Facial Images , 2006 .

[6]  Kyandoghere Kyamakya,et al.  A Driver State Detection System—Combining a Capacitive Hand Detection Sensor With Physiological Sensors , 2017, IEEE Transactions on Instrumentation and Measurement.

[7]  Tong Zhang,et al.  Spatial–Temporal Recurrent Neural Network for Emotion Recognition , 2017, IEEE Transactions on Cybernetics.

[8]  Manolis Tsiknakis,et al.  Stress and anxiety detection using facial cues from videos , 2017, Biomed. Signal Process. Control..

[9]  Maarten A. Hogervorst,et al.  A new paradigm to induce mental stress: the Sing-a-Song Stress Test (SSST) , 2014, Front. Neurosci..

[10]  Mauricio A. Álvarez,et al.  Feature selection for multimodal emotion recognition in the arousal-valence space , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[11]  Wenbo Luo,et al.  Emotional Noun Processing: An ERP Study with Rapid Serial Visual Presentation , 2015, PloS one.

[12]  Rashima Mahajan,et al.  Emotion Recognition via EEG Using Neural Network Classifier , 2018 .

[13]  Olga Sourina,et al.  Real-Time EEG-Based Emotion Recognition and Its Applications , 2011, Trans. Comput. Sci..

[14]  Mohd Nasir Taib,et al.  Learners’ Learning Style Classification related to IQ and Stress based on EEG , 2011 .

[15]  Qisong Wang,et al.  Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition , 2016, Comput. Biol. Medicine.

[16]  Yongyuth Pongsupap,et al.  Analysis of heart rate variability during meditation by a pattern recognition method , 2011, 2011 Computing in Cardiology.

[17]  G. Butler,et al.  Definitions of stress. , 1993, Occasional paper.

[18]  Brock Bastian,et al.  The Discrete Emotions Questionnaire: A New Tool for Measuring State Self-Reported Emotions , 2016, PloS one.

[19]  Hesham Ahmed Hefny,et al.  EEG-Based Emotion Recognition Using a Wrapper-Based Feature Selection Method , 2017, AISI.

[20]  Emre Ertin,et al.  Continuous inference of psychological stress from sensory measurements collected in the natural environment , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[21]  Y. Okada,et al.  Wearable ECG recorder with acceleration sensors for monitoring daily stress: Office work simulation study , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[22]  S. Folkman,et al.  Stress, appraisal, and coping , 1974 .

[23]  João Paulo da Silva Cunha,et al.  Droid Jacket: Using an Android based smartphone for team monitoring , 2011, 2011 7th International Wireless Communications and Mobile Computing Conference.

[24]  Zhenqi Li,et al.  Emotion Recognition from EEG Using RASM and LSTM , 2017, ICIMCS.

[25]  J. Russell.  A circumplex model of affect. , 1980 .

[26]  Aamir Saeed Malik,et al.  Machine Learning Framework for the Detection of Mental Stress at Multiple Levels , 2017, IEEE Access.

[27]  Dinesh Singh,et al.  EEG Based Emotion Classification Mechanism in BCI , 2018 .

[28]  Deepak Khosla,et al.  Spatial mislocalization of EEG electrodes – effects on accuracy of dipole estimation , 1999, Clinical Neurophysiology.

[29]  Masayuki Numao,et al.  Familiarity effects in EEG-based emotion recognition , 2016, Brain Informatics.

[30]  Jean M. Vettel,et al.  Different profiles of decision making and physiology under varying levels of stress in trained military personnel. , 2018, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[31]  Nasreen Badruddin,et al.  Towards multilevel mental stress assessment using SVM with ECOC: an EEG approach , 2017, Medical & Biological Engineering & Computing.

[32]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[33]  Nico Pallamin,et al.  Emotion Recognition Using Physiological Signals: Laboratory vs. Wearable Sensors , 2017, AHFE.

[34]  Peter Salovey,et al.  Rethinking stress: the role of mindsets in determining the stress response. , 2013, Journal of personality and social psychology.

[35]  Javad Frounchi,et al.  Wavelet-based emotion recognition system using EEG signal , 2017, Neural Computing and Applications.

[36]  Muhammad Zubair,et al.  EEG Based Classification of Human Emotions Using Discrete Wavelet Transform , 2018 .

[37]  Rossana Castaldo,et al.  Ultra-Short Entropy for Mental Stress Detection , 2018, IFMBE Proceedings.

[38]  Murugappan Murugappan,et al.  ECG Signal Denoising Using Wavelet Thresholding Techniques in Human Stress Assessment , 2012 .

[39]  Adrian Basarab,et al.  Using smart offices to predict occupational stress , 2018, International Journal of Industrial Ergonomics.

[40]  S. Hsieh,et al.  Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns , 2014, PloS one.

[41]  W. M. Millar,et al.  Emotion and Blood-Pressure , 1959 .

[42]  Léon J. M. Rothkrantz,et al.  Emotion recognition using brain activity , 2008, CompSysTech.

[43]  Sebastian A.F. Stehlin,et al.  EEG with a reduced number of electrodes: Where to detect and how to improve visually, auditory and somatosensory evoked potentials , 2018 .

[44]  Mohammad H. Mahoor,et al.  A wavelet-based approach to emotion classification using EDA signals , 2018, Expert Syst. Appl..

[45]  Ahmad R. Sharafat,et al.  Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals , 2015, Comput. Methods Programs Biomed..

[46]  Luca Citi,et al.  Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics , 2014, Scientific Reports.

[47]  Wan-Young Chung,et al.  Wearable Glove-Type Driver Stress Detection Using a Motion Sensor , 2017, IEEE Transactions on Intelligent Transportation Systems.

[48]  P. Sleight,et al.  Effects of controlled breathing, mental activity and mental stress with or without verbalization on heart rate variability. , 2000, Journal of the American College of Cardiology.

[49]  Nicu Sebe,et al.  MULTIMODAL EMOTION RECOGNITION , 2005 .

[50]  Jacek M. Leski,et al.  Validation of Emotiv EPOC+ for extracting ERP correlates of emotional face processing , 2018 .

[51]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[52]  Xin Zhao,et al.  Study on an effective cross-stimulus emotion recognition model using EEGs based on feature selection and support vector machine , 2016, International Journal of Machine Learning and Cybernetics.

[53]  Aydin Akan,et al.  Emotion recognition from EEG signals by using multivariate empirical mode decomposition , 2018, Pattern Analysis and Applications.

[54]  Dezhong Yao,et al.  How do reference montage and electrodes setup affect the measured scalp EEG potentials? , 2018, Journal of neural engineering.

[55]  M. al’Absi,et al.  Stress and Addiction: Biological and Psychological Mechanisms. , 2007 .

[56]  R. Rahe,et al.  The Brief Stress and Coping Inventory: A Useful Stress Management Instrument , 2002 .

[57]  H. Selye.  The Stress of Life , 1958 .

[58]  Tamás D. Gedeon,et al.  Modeling a stress signal , 2014, Appl. Soft Comput..

[59]  Khaled Shaalan,et al.  Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2017, AISI 2017, Cairo, Egypt, September 9-11, 2017 , 2018, AISI.

[60]  Chi Zhang,et al.  Emotion Recognition from EEG Signals Using Multidimensional Information in EMD Domain , 2017, BioMed research international.

[61]  M. Enoch,et al.  Genetic and Environmental Influences on the Development of Alcoholism , 2006, Annals of the New York Academy of Sciences.

[62]  Rosa Gil,et al.  Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way , 2016, Front. Comput. Neurosci..

[63]  Eloy Irigoyen,et al.  An enhanced fuzzy algorithm based on advanced signal processing for identification of stress , 2018, Neurocomputing.

[64]  A. Moors.  Theories of emotion causation: A review , 2009 .

[65]  H. Becher,et al.  Absolute Risk and Loss-of-Lifetime Estimates for Quantitative Risk Assessment , 1998 .

[66]  Shin-Dug Kim,et al.  Electroencephalography Based Fusion Two-Dimensional (2D)-Convolution Neural Networks (CNN) Model for Emotion Recognition System , 2018, Sensors.

[67]  Loïc Kessous,et al.  Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis , 2010, Journal on Multimodal User Interfaces.

[68]  Wee Ser,et al.  Discrete Wavelet Transform coefficients for emotion recognition from EEG signals , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[69]  José Manuel Pastor,et al.  Emotion Detection in Ageing Adults from Physiological Sensors , 2015, ISAmI.

[70]  Xue Wang,et al.  Wearable biosensor network enabled multimodal daily-life emotion recognition employing reputation-driven imbalanced fuzzy classification , 2017 .

[71]  Mi Li,et al.  Emotion recognition from multichannel EEG signals using K-nearest neighbor classification , 2018, Technology and health care : official journal of the European Society for Engineering and Medicine.

[72]  Adrian Basarab,et al.  Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review , 2016, J. Biomed. Informatics.

[73]  Moon Inder Singh,et al.  Development of a real time emotion classifier based on evoked EEG , 2017 .

[74]  Arun Kumar Sangaiah,et al.  An Approach to EEG Based Emotion Recognition and Classification Using Kernel Density Estimation , 2015 .

[75]  Maryam Shojaei Baghini,et al.  Mental stress assessment - a comparison between HRV based and respiration based techniques , 2015, 2015 Computing in Cardiology Conference (CinC).

[76]  Vinod Chandran,et al.  Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors , 2018, Expert Syst. Appl..

[77]  A. Goshvarpour,et al.  Fusion of heart rate variability and pulse rate variability for emotion recognition using lagged poincare plots , 2017, Australasian Physical & Engineering Sciences in Medicine.

[78]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[79]  M. Enoch,et al.  Pharmacogenomics of Alcohol Response and Addiction , 2003, American journal of pharmacogenomics : genomics-related research in drug development and clinical practice.

[80]  Mariette Awad,et al.  Anytime multipurpose emotion recognition from EEG data using a Liquid State Machine based framework , 2018, Artif. Intell. Medicine.

[81]  Fernando Seoane,et al.  Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones , 2015, Sensors.

[82]  Stefan Wermter,et al.  Multimodal emotional state recognition using sequence-dependent deep hierarchical features , 2015, Neural Networks.

[83]  Fabien Ringeval,et al.  An array of physical sensors and an adaptive regression strategy for emotion recognition in a noisy scenario , 2017 .

[84]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[85]  Michael D. Robinson,et al.  Measures of emotion: A review , 2009, Cognition & emotion.

[86]  Sidney K. D'Mello,et al.  A Review and Meta-Analysis of Multimodal Affect Detection Systems , 2015, ACM Comput. Surv..

[87]  Christian Mühl,et al.  Design and Validation of a Mental and Social Stress Induction Protocol - Towards Load-invariant Physiology-based Stress Detection , 2014, PhyCS.

[88]  Riccardo Bernardini,et al.  Driver’s stress detection using Skin Potential Response signals , 2018, Measurement.

[89]  Yaacob Sazali,et al.  Classification of human emotion from EEG using discrete wavelet transform , 2010 .

[90]  Yongik Yoon,et al.  Multi-level assessment model for wellness service based on human mental stress level , 2017, Multimedia Tools and Applications.

[91]  Marc Thomas Philipp Adam,et al.  Blended Emotion Detection for Decision Support , 2015, IEEE Transactions on Human-Machine Systems.

[92]  Ricardo Gutierrez-Osuna,et al.  Development and Evaluation of an Ambulatory Stress Monitor Based on Wearable Sensors , 2012, IEEE Transactions on Information Technology in Biomedicine.

[93]  Houtan Jebelli,et al.  EEG-based workers' stress recognition at construction sites , 2018, Automation in Construction.

[94]  M. Teplan.  FUNDAMENTALS OF EEG MEASUREMENT , 2002 .

[95]  Navjot Bhullar,et al.  The Assessing Emotions Scale. , 2009 .

[96]  Dae SeokLee,et al.  Stress Events Detection of Driver by Wearable Glove System , 2017, IEEE Sensors Journal.

[97]  Bao-Liang Lu,et al.  Emotion classification based on gamma-band EEG , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[98]  Julien Penders,et al.  Trapezius muscle EMG as predictor of mental stress , 2013, TECS.

[99]  K. Maghooli,et al.  Modeling the 2D space of emotions based on the poincare plot of heart rate variability signal , 2018 .

[100]  Sheldon Cohen,et al.  PERCEIVED STRESS SCALE , 2014 .

[101]  Andrew Begel,et al.  Using psycho-physiological measures to assess task difficulty in software development , 2014, ICSE.

[102]  Ali Momennezhad,et al.  EEG-based emotion recognition utilizing wavelet coefficients , 2018, Multimedia Tools and Applications.

[103]  P. Kleinginna,et al.  A categorized list of emotion definitions, with suggestions for a consensual definition , 1981 .

[104]  Hyo Jong Lee,et al.  A novel feature extraction method based on late positive potential for emotion recognition in human brain signal patterns , 2016, Comput. Electr. Eng..

[105]  Nicole Novielli,et al.  Emotion detection using noninvasive low cost sensors , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[106]  Isabelle Bichindaritz,et al.  Feature Selection and Machine Learning Based Multilevel Stress Detection from ECG Signals , 2017 .

[107]  Klaus R. Scherer,et al.  Emotion, Development, and Self-Organization: Emotions as Episodes of Subsystem Synchronization Driven by Nonlinear Appraisal Processes , 2000 .

[108]  D. Tucker,et al.  EEG source localization: Sensor density and head surface coverage , 2015, Journal of Neuroscience Methods.

[109]  Ingo Siegert,et al.  Using a PCA-based dataset similarity measure to improve cross-corpus emotion recognition , 2018, Comput. Speech Lang..

[110]  Arnab Bag,et al.  Effects of emotion on physiological signals , 2016, 2016 IEEE Annual India Conference (INDICON).

[111]  Sheng Hong,et al.  Classification of the emotional stress and physical stress using signal magnification and canonical correlation analysis , 2018, Pattern Recognit..

[112]  D. Jude Hemanth,et al.  Brain signal based human emotion analysis by circular back propagation and Deep Kohonen Neural Networks , 2018, Comput. Electr. Eng..

[113]  Fadel Adib,et al.  Emotion recognition using wireless signals , 2016, MobiCom.

[114]  L. Quintero,et al.  An approach to emotion recognition in single-channel EEG signals: a mother child interaction , 2016 .

[115]  Atefeh Goshvarpour,et al.  An accurate emotion recognition system using ECG and GSR signals and matching pursuit method , 2017, Biomedical journal.

[116]  Niraj K. Jha,et al.  Keep the Stress Away with SoDA: Stress Detection and Alleviation System , 2017, IEEE Transactions on Multi-Scale Computing Systems.

[117]  Jon D. Morris.  Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[118]  Klaus Wehrle,et al.  EEG-based automatic emotion recognition: Feature extraction, selection and classification methods , 2016, 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom).

[119]  Chao Li,et al.  Analysis of physiological for emotion recognition with the IRS model , 2016, Neurocomputing.

[120]  Karel Brookhuis,et al.  Handbook of Human Factors and Ergonomics Methods , 2009 .

[121]  Yong Zhang,et al.  EEG-based classification of emotions using empirical mode decomposition and autoregressive model , 2018, Multimedia Tools and Applications.

[122]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[123]  P. Lang.  International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .

[124]  Felipe Trujillo-Romero,et al.  Multimodal emotion recognition with evolutionary computation for human-robot interaction , 2016, Expert Syst. Appl..

[125]  K. M. Ravi Kumar,et al.  EEG Based Emotion Recognition Using Wavelets and Neural Networks Classifier , 2018 .

[126]  Olga Sourina,et al.  EEG-Based Mental Workload and Stress Monitoring of Crew Members in Maritime Virtual Simulator , 2018, Trans. Comput. Sci..

[127]  Nitin Kumar,et al.  Bispectral Analysis of EEG for Emotion Recognition , 2015, IHCI.

[128]  Tamás D. Gedeon,et al.  Objective measures, sensors and computational techniques for stress recognition and classification: A survey , 2012, Comput. Methods Programs Biomed..

[129]  Hyeoncheol Kim,et al.  Emotion extraction based on multi bio-signal using back-propagation neural network , 2018, Multimedia Tools and Applications.

[130]  Theerasak Chanwimalueang,et al.  Discrimination of emotional states from scalp- and intracranial EEG using multiscale Rényi entropy , 2017, PloS one.