Objective measures, sensors and computational techniques for stress recognition and classification: A survey

Stress is a major growing concern in our day and age adversely impacting both individuals and society. Stress research has a wide range of benefits from improving personal operations, learning, and increasing work productivity to benefiting society - making it an interesting and socially beneficial area of research. This survey reviews sensors that have been used to measure stress and investigates techniques for modelling stress. It discusses non-invasive and unobtrusive sensors for measuring computed stress, a term we coin in the paper. Sensors that do not impede everyday activities that could be used by those who would like to monitor stress levels on a regular basis (e.g. vehicle drivers, patients with illnesses linked to stress) is the focus of the discussion. Computational techniques have the capacity to determine optimal sensor fusion and automate data analysis for stress recognition and classification. Several computational techniques have been developed to model stress based on techniques such as Bayesian networks, artificial neural networks, and support vector machines, which this survey investigates. The survey concludes with a summary and provides possible directions for further computational stress research.

[1]  J. Kabat-Zinn,et al.  Three-year follow-up and clinical implications of a mindfulness meditation-based stress reduction intervention in the treatment of anxiety disorders. , 1995, General hospital psychiatry.

[2]  Rahul Banerjee,et al.  Detection of fatigue of vehicular driver using skin conductance and oximetry pulse: a neural network approach , 2009, iiWAS.

[3]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[4]  Andrew Steptoe,et al.  Impaired cardiovascular recovery following stress predicts 3-year increases in blood pressure , 2005, Journal of hypertension.

[5]  Jennifer A. Healey,et al.  Wearable and automotive systems for affect recognition from physiology , 2000 .

[6]  Suzanne C Danhauer,et al.  Music as Therapy , 2005, Southern medical journal.

[7]  John H. L. Hansen,et al.  Frequency band analysis for stress detection using a teager energy operator based feature , 2002, INTERSPEECH.

[8]  Ulf Lundberg,et al.  Stress, subjective and objective health. , 2006 .

[9]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[10]  Rosalind W. Picard,et al.  Modeling drivers' speech under stress , 2003, Speech Commun..

[11]  Thomas Steckler,et al.  Handbook of Stress and the Brain Part 1: The Neurobiology of Stress , 2005 .

[12]  Kristina Höök,et al.  License to chill!: how to empower users to cope with stress , 2008, NordiCHI.

[13]  Lizawati Salahuddin,et al.  Detection of Acute Stress by Heart Rate Variability Using a Prototype Mobile ECG Sensor , 2006 .

[14]  D. Hennessy,et al.  Traffic congestion, driver stress, and driver aggression , 1999 .

[15]  Dimitris N. Metaxas,et al.  Optical computer recognition of facial expressions associated with stress induced by performance demands. , 2005, Aviation, space, and environmental medicine.

[16]  Mohd Nasir Taib,et al.  Initial investigation of human physical stress level using brainwaves , 2009, 2009 IEEE Student Conference on Research and Development (SCOReD).

[17]  Léon J. M. Rothkrantz,et al.  Voice Stress Analysis , 2004, TSD.

[18]  John L. Sibert,et al.  Heart rate variability: indicator of user state as an aid to human-computer interaction , 1998, CHI.

[19]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[20]  J E Schwartz,et al.  Environmental influences on blood pressure and the role of job strain. , 1996, Journal of hypertension. Supplement : official journal of the International Society of Hypertension.

[21]  Claudia-Adina Dragos,et al.  Stable and optimal fuzzy control of a laboratory Antilock Braking System , 2010, 2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[22]  William B. Langdon,et al.  Measuring facial emotional expressions using genetic programming , 2002 .

[23]  S. S. Reisman,et al.  Measurement of physiological stress , 1997, Proceedings of the IEEE 23rd Northeast Bioengineering Conference.

[24]  Cary L. Cooper,et al.  Preventing stress, improving productivity: European case studies in the workplace , 1999 .

[25]  Kevin R Ward,et al.  Emotional distress, coping and adjustment in family members of neuroscience intensive care unit patients. , 2009, Journal of psychosomatic research.

[26]  Imad H. Elhajj,et al.  Multi level SVM for subject independent agitation detection , 2009, 2009 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[27]  E. Braunwald,et al.  Control of Heart Rate by the Autonomic Nervous System: Studies in Man on the Interrelation Between Baroreceptor Mechanisms and Exercise , 1966, Circulation research.

[28]  James M. Ferryman,et al.  A Combined Bayesian Markovian Approach for Behaviour Recognition , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[29]  Anton Nijholt,et al.  Manipulating Stress and Cognitive Load in Conversational Interactions with a Multimodal System for Crisis Management Support , 2009, COST 2102 Training School.

[30]  Glenn F. Wilson,et al.  Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks , 2003, Hum. Factors.

[31]  Say Wei Foo,et al.  Speech emotion recognition using hidden Markov models , 2003, Speech Commun..

[32]  Murray B. Stein,et al.  Functional Activation and Neural Networks in Women with Posttraumatic Stress Disorder Related to Intimate Partner Violence , 2008, Biological Psychiatry.

[33]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[34]  Norbert Stoll,et al.  Fuzzy Techniques for Subjective Workload-Score Modeling Under Uncertainties , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  David A. Morilak,et al.  Role of brain norepinephrine in the behavioral response to stress , 2005, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[36]  A. K. Blangsted,et al.  The effect of mental stress on heart rate variability and blood pressure during computer work , 2004, European Journal of Applied Physiology.

[37]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[38]  Christian Müller Speaker Classification II, Selected Projects , 2007, Speaker Classification.

[39]  Tanja Schultz,et al.  Identifying User State Using Electroencephalographic Data , 2005 .

[40]  Gaetano Borriello,et al.  Guest Editors' Introduction: Pervasive Computing in Sports Technologies , 2005, IEEE Pervasive Comput..

[41]  Knut Engedal,et al.  High score on the Relative Stress Scale, a marker of possible psychiatric disorder in family carers of patients with dementia , 2007, International journal of geriatric psychiatry.

[42]  Zhiwei Zhu,et al.  A Real-Time Human Stress Monitoring System Using Dynamic Bayesian Network , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[43]  Andrew Liu,et al.  MODELING AND PREDICTION OF HUMAN DRIVER BEHAVIOR , 2001 .

[44]  Luc Van Gool,et al.  Recognizing emotions expressed by body pose: A biologically inspired neural model , 2008, Neural Networks.

[45]  Constantine Kotropoulos,et al.  Emotional speech recognition: Resources, features, and methods , 2006, Speech Commun..

[46]  Wilfrid Jänig,et al.  Characteristics of function-specific pathways in the sympathetic nervous system , 1992, Trends in Neurosciences.

[47]  Ahmad R. Hariri,et al.  Facial Expressions of Emotion Reveal Neuroendocrine and Cardiovascular Stress Responses , 2007, Biological Psychiatry.

[48]  Matthias Weippert,et al.  Fuzzy Evaluation of Heart Rate Signals for Mental Stress Assessment , 2007, IEEE Transactions on Fuzzy Systems.

[49]  Toshio Tsuji,et al.  Evaluation of heart rate variability by using wavelet transform and a recurrent neural network , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[50]  L. R. John,et al.  Quantifying Mental Relaxation with EEG for use in Computer Games , 2006, International Conference on Internet Computing.

[51]  P. Jin,et al.  Efficacy of Tai Chi, brisk walking, meditation, and reading in reducing mental and emotional stress. , 1992, Journal of psychosomatic research.

[52]  S. A. Hosseini,et al.  Emotional Stress Recognition System Using EEG and Psychophysiological Signals: Using New Labelling Process of EEG Signals in Emotional Stress State , 2010, 2010 International Conference on Biomedical Engineering and Computer Science.

[53]  Brian McDonald,et al.  Intelligent Biofeedback using an Immersive Competitive Environment , 2001 .

[54]  Zulfikar Dharmawan,et al.  Analysis of Computer Games Player Stress Level Using EEG Data , 2007 .

[55]  John H. L. Hansen,et al.  Nonlinear feature based classification of speech under stress , 2001, IEEE Trans. Speech Audio Process..

[56]  Gregory F. Lewis,et al.  Accuracy of the StressEraser® in the Detection of Cardiac Rhythms , 2008, Applied psychophysiology and biofeedback.

[57]  Armando Barreto,et al.  Non-intrusive Physiological Monitoring for Automated Stress Detection in Human-Computer Interaction , 2007, ICCV-HCI.

[58]  Yew-Soon Ong,et al.  Advances in Natural Computation, First International Conference, ICNC 2005, Changsha, China, August 27-29, 2005, Proceedings, Part I , 2005, ICNC.

[59]  L. H. Miller,et al.  Replicability of two GSR indices as a function of stress and cognitive activity. , 1965, Journal of personality and social psychology.

[60]  Armando Barreto,et al.  Stress Recognition Using Non-invasive Technology , 2006, FLAIRS.

[61]  S. Segerstrom,et al.  Psychological stress and the human immune system: a meta-analytic study of 30 years of inquiry. , 2004, Psychological bulletin.

[62]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[63]  É. Labbé,et al.  Coping with Stress: The Effectiveness of Different Types of Music , 2007, Applied psychophysiology and biofeedback.

[64]  John J. Grieco,et al.  Evaluation of Voice Stress Analysis Technology , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[65]  F. Braceland THE STRESS OF LIFE , 1976 .

[66]  P. Greenberg,et al.  Significance of changes in R wave amplitude during treadmill stress testing: angiographic correlation. , 1978, The American journal of cardiology.

[67]  É. Labbé,et al.  Health Psychology: A Psychobiological Perspective , 1986 .

[68]  R. Suganya,et al.  Data Mining Concepts and Techniques , 2010 .

[69]  U. Rajendra Acharya,et al.  Heart rate variability: a review , 2006, Medical and Biological Engineering and Computing.

[70]  M S Nieminen,et al.  Left ventricular wall stresses and wall stress-mass-heart rate products in hypertensive patients with electrocardiographic left ventricular hypertrophy: the LIFE study. Losartan Intervention For Endpoint reduction in hypertension. , 2000, Journal of hypertension.

[71]  L. Sornmo,et al.  Analysis of Heart Rate Variability Using Time-Varying Frequency Bands Based on Respiratory Frequency , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[72]  L. J. M. Rothkrantz,et al.  DETECTING STRESS USING EYE BLINKS AND BRAIN ACTIVITY FROM EEG SIGNALS , 2009 .

[73]  Ying Gao,et al.  Pupil diameter measurements: untapped potential to enhance computer interaction for eye tracker users? , 2008, Assets '08.

[74]  Mohd Nasir Taib,et al.  Evaluation of human stress using EEG Power Spectrum , 2010, 2010 6th International Colloquium on Signal Processing & its Applications.

[75]  Munna Khan,et al.  A Review of Measurement and Analysis of Heart Rate Variability , 2009, 2009 International Conference on Computer and Automation Engineering.

[76]  Maria E. Jabon,et al.  Facial expression analysis for predicting unsafe driving behavior , 2011, IEEE Pervasive Computing.

[77]  Björn W. Schuller,et al.  Hidden Markov model-based speech emotion recognition , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[78]  Sharon M. Walter,et al.  Investigation and Evaluation of Voice Stress Analysis Technology , 2001 .

[79]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[80]  J. Sexton,et al.  Error, Stress, and Teamwork in Medicine and Aviation: Cross Sectional Surveys , 2001 .

[81]  Kaliappan Gopalan On the effect of stress on certain modulation parameters of speech , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[82]  Chris Baber,et al.  Towards a definition and working model of stress and its effects on speech , 1996, Speech Commun..

[83]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[84]  Veikko Surakka,et al.  Pupil size variation as an indication of affective processing , 2003, Int. J. Hum. Comput. Stud..

[85]  Lionel Tarassenko Signal processing methods for heart rate variability , 2002 .

[86]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[87]  Trevor Jackson,et al.  Traffic congestion , 2003, BMJ : British Medical Journal.

[88]  Jennifer Healey,et al.  SmartCar: detecting driver stress , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[89]  Shogo Nishida,et al.  A Method for Selecting Facial Expression based on Emotions and Its Application for Text Reading , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[90]  Günther Palm,et al.  Emotion Recognition from Speech: Stress Experiment , 2008, LREC.

[91]  Yoshio Nakamura,et al.  Heart rate variability, trait anxiety, and perceived stress among physically fit men and women. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[92]  Franck Davoine,et al.  A solution for facial expression representation and recognition , 2002, Signal Process. Image Commun..

[93]  Tao Lin,et al.  Do physiological data relate to traditional usability indexes? , 2005, OZCHI.

[94]  John H. L. Hansen,et al.  Speech Under Stress: Analysis, Modeling and Recognition , 2007, Speaker Classification.

[95]  Andrew Sears,et al.  Automated stress detection using keystroke and linguistic features: An exploratory study , 2009, Int. J. Hum. Comput. Stud..

[96]  Sun Kook Yoo,et al.  Neural Network Based Emotion Estimation Using Heart Rate Variability and Skin Resistance , 2005, ICNC.

[97]  B. McEwen,et al.  Stress and cognitive function , 1995, Current Opinion in Neurobiology.

[98]  Anton Nijholt,et al.  Development of Multimodal Interfaces: Active Listening and Synchrony, Second COST 2102 International Training School, Dublin, Ireland, March 23-27, 2009, Revised Selected Papers , 2010, COST 2102 Training School.

[99]  Y Fukuoka,et al.  Chronic stress evaluation using neural networks. , 2000, IEEE engineering in medicine and biology magazine : the quarterly magazine of the Engineering in Medicine & Biology Society.

[100]  Mika P. Tarvainen,et al.  Software for advanced HRV analysis , 2004, Comput. Methods Programs Biomed..

[101]  Louise Lemyre,et al.  Measuring psychological stress. Concept, model, and measurement instrument in primary care research. , 2003, Canadian family physician Medecin de famille canadien.

[102]  P. Stone,et al.  Use of skin conductance changes during mental stress testing as an index of autonomic arousal in cardiovascular research. , 1994, American heart journal.

[103]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[104]  N. Mendell,et al.  Hostility and cardiovascular reactivity to stress in women and men. , 1989, Psychosomatic medicine.

[105]  W. S. Peavler,et al.  Pupil size, information overload, and performance differences. , 1974, Psychophysiology.

[106]  H. Selye The Stress of Life , 1958 .

[107]  Karl Kirsch,et al.  Voice monitoring to measure emotional load during short-term stress , 2002, European Journal of Applied Physiology.

[108]  Carl W. Sem-Jacobsen ELECTROENCEPHALOGRAPHIC STUDY OF PILOT STRESSES IN FLIGHT , 1961 .

[109]  P. Hassmén,et al.  Psychophysiological stress and emg activity of the trapezius muscle , 1994, International journal of behavioral medicine.

[110]  A. H. Roscoe Assessing pilot workload. Why measure heart rate, HRV and respiration? , 1992, Biological Psychology.

[111]  Léon J. M. Rothkrantz,et al.  Emotion recognition using brain activity , 2008, CompSysTech.

[112]  Frank E. Ritter,et al.  Measuring the Effect of Dental Work as a Stressor on Cognition , 2005 .

[113]  Alex Pentland,et al.  Modeling and Prediction of Human Behavior , 1999, Neural Computation.

[114]  A. H. Roscoe,et al.  Assessing Pilot Workload , 1978 .

[115]  J. Movellan,et al.  Human and computer recognition of facial expressions of emotion , 2007, Neuropsychologia.

[116]  Stephen P. Boyd,et al.  Fuzzy Filtering for Physiological Signal Analysis , 2010 .

[117]  A. Barreto,et al.  Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[118]  B. Hjorth EEG analysis based on time domain properties. , 1970, Electroencephalography and clinical neurophysiology.