Classification of the emotional stress and physical stress using signal magnification and canonical correlation analysis

The emotional state of a person can be obtained by extracting individual stress index, which provides useful information to health field.This paper directs the frontier of industrializing stress recognition by focusing on establishing a set of non-contact imaging based classifications for emotional stress (ES) and Physical stress (PS).The proposed algorithm successfully extracts weak signal from thermal imaging for stress classification.The classification algorithm is effective with accuracy as high as 90%. This study can lead to a practical system for the noninvasive assessment of stress states for practical applications. In affective computing, stress recognition mainly focuses on the relation of stress and photoelectric information. Researchers have used artificial intelligence to determine stress and computer identification channels. However, in applications such as health and security, Emotional stress (ES) information is usually to be alongside physical stress (PS) information, making it urgent to classify ES and PS. The thermal signals of ES and PS have yet to be classified, for which, signal amplification is offered. In this study, we propose a classification algorithm based on signal amplification and correlation analysis called Eulerian magnification-canonical correlation analysis. This signal amplification algorithm expands the signals of ES and PS in different frequency domains. Sparse coding and canonical correlation analysis then fuse the original signal and its amplified features. The extracted entropy features are used to train the correlation weight between ES and PS, which formulates stress classifications. The new classification method achieves an accuracy rate of 90%. This study can lead to a practical system for the noninvasive assessment of stress states for health or security applications.

[1]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[2]  Lapitskiĭ Fg,et al.  Sudden death of apparently healthy adolescents under the normal physical stress of sport activities , 1970 .

[3]  P Hjemdahl,et al.  Effects of mental and physical stress on platelet function in patients with stable angina pectoris and healthy controls. , 1997, European heart journal.

[4]  P D Drummond,et al.  The effect of adrenergic blockade on blushing and facial flushing. , 1997, Psychophysiology.

[5]  David Sander,et al.  Thermal Analysis of Facial Muscles Contractions , 2011, IEEE Transactions on Affective Computing.

[6]  Ioannis T. Pavlidis,et al.  Thermal image analysis for anxiety detection , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[7]  Ioannis T. Pavlidis,et al.  Description and clinical studies of a device for the instantaneous detection of office-place stress. , 2009, Work.

[8]  A. Merla,et al.  Mother and child in synchrony: Thermal facial imprints of autonomic contagion , 2012, Biological Psychology.

[9]  Nikolaos G. Bourbakis,et al.  Effects of Emotional Visual Scenes on the Ability to Decode Emotional Melodies , 2016, 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI).

[10]  Kevin A. Johnson,et al.  Detecting Deception Using Functional Magnetic Resonance Imaging , 2005, Biological Psychiatry.

[11]  Surya Sumpeno,et al.  Simulation multi behavior NPCs in fire evacuation using emotional behavior tree , 2016, 2016 International Seminar on Application for Technology of Information and Communication (ISemantic).

[12]  A. Merla,et al.  Thermal signature of fear conditioning in mild post traumatic stress disorder , 2014, Neuroscience.

[13]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[14]  V. Gallese,et al.  Thermal infrared imaging in psychophysiology: Potentialities and limits , 2014, Psychophysiology.

[15]  H Selye,et al.  Confusion and controversy in the stress field. , 1975, Journal of human stress.

[16]  Munaf Rashid,et al.  Human emotion recognition from videos using spatio-temporal and audio features , 2012, The Visual Computer.

[17]  Qingshan Liu,et al.  Learning Multiscale Active Facial Patches for Expression Analysis , 2015, IEEE Transactions on Cybernetics.

[18]  Mohan M. Trivedi,et al.  Toward Privacy-Protecting Safety Systems for Naturalistic Driving Videos , 2014, IEEE Transactions on Intelligent Transportation Systems.

[19]  Amita M Pardeshi,et al.  Comparison of Anthropometric Parameters and Blood Pressure Changes in Response to Physical Stress Test in Normotensive Subjects with or Without Family History of Hypertension. , 2016, Indian journal of physiology and pharmacology.

[20]  Matthew L. Jensen,et al.  Detecting Concealment of Intent in Transportation Screening: A Proof of Concept , 2009, IEEE Transactions on Intelligent Transportation Systems.

[21]  Ioannis Pavlidis,et al.  Seeing through the face of deception , 2002, Nature.

[22]  Nikolaos G. Bourbakis,et al.  Extracting and Associating Meta-features for Understanding People’s Emotional Behaviour: Face and Speech , 2011, Cognitive Computation.

[23]  Tao Chen,et al.  Uploader Intent for Online Video: Typology, Inference, and Applications , 2015, IEEE Transactions on Multimedia.

[24]  Shrikanth S. Narayanan,et al.  Distributional Semantic Models for Affective Text Analysis , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[25]  Elizabeth McCay Experience of emotional stress and resilience in street-involved youth: the need for early mental health intervention. , 2011, Healthcare quarterly.

[26]  Jukka Kortelainen,et al.  Classifier-based learning of nonlinear feature manifold for visualization of emotional speech prosody , 2013, IEEE Transactions on Affective Computing.

[27]  Rie Nakanishi,et al.  Facial skin temperature decreases in infants with joyful expression. , 2008, Infant behavior & development.

[28]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Michael Anbar,et al.  Assessment of Physiologic and Pathologic Radiative Heat Dissipation Using Dynamic Infrared Imaging , 2002, Annals of the New York Academy of Sciences.

[30]  Enzo Pasquale Scilingo,et al.  Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability , 2015, IEEE Transactions on Affective Computing.

[31]  Chang Dong Yoo,et al.  Loss-Scaled Large-Margin Gaussian Mixture Models for Speech Emotion Classification , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[32]  C. Kirschbaum,et al.  The 'Trier Social Stress Test'--a tool for investigating psychobiological stress responses in a laboratory setting. , 1993, Neuropsychobiology.

[33]  Mark A. Clements,et al.  Multimodal Affect Classification at Various Temporal Lengths , 2015, IEEE Transactions on Affective Computing.

[34]  W. Cannon The Wisdom of the Body , 1932 .

[35]  F. Plessow,et al.  Classification Criteria for Distinguishing Cortisol Responders From Nonresponders to Psychosocial Stress: Evaluation of Salivary Cortisol Pulse Detection in Panel Designs , 2013, Psychosomatic medicine.

[36]  Rama Chellappa,et al.  Structure-Preserving Sparse Decomposition for Facial Expression Analysis , 2014, IEEE Transactions on Image Processing.

[37]  Wanhui Wen,et al.  Emotion Recognition Based on Multi-Variant Correlation of Physiological Signals , 2014, IEEE Transactions on Affective Computing.

[38]  A. Raut,et al.  Effect of Withania somnifera on physical and cardiovascular performance induced by physical stress in healthy human volunteers , 2016 .

[39]  I. Pavlidis,et al.  Fast by Nature - How Stress Patterns Define Human Experience and Performance in Dexterous Tasks , 2012, Scientific Reports.

[40]  Jun Li,et al.  Robust Representation and Recognition of Facial Emotions Using Extreme Sparse Learning , 2015, IEEE Transactions on Image Processing.

[41]  A. Merla,et al.  Mom feels what her child feels: thermal signatures of vicarious autonomic response while watching children in a stressful situation , 2013, Front. Hum. Neurosci..

[42]  Liming Chen,et al.  Muscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition , 2015, IEEE Transactions on Multimedia.

[43]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[44]  C.J. Cohen,et al.  A Surveillance System for the Recognition of Intent within Individuals and Crowds , 2008, 2008 IEEE Conference on Technologies for Homeland Security.

[45]  Goldie Nejat,et al.  Promoting Interactions Between Humans and Robots Using Robotic Emotional Behavior , 2016, IEEE Transactions on Cybernetics.

[46]  Qing Chen,et al.  Primary immune surveillance: some like it hot , 2007, Journal of Molecular Medicine.

[47]  João Barros,et al.  A Mobile Sensing Approach to Stress Detection and Memory Activation for Public Bus Drivers , 2015, IEEE Transactions on Intelligent Transportation Systems.

[48]  Vittorio Gallese,et al.  The Autonomic Signature of Guilt in Children: A Thermal Infrared Imaging Study , 2013, PloS one.

[49]  Akio Nozawa,et al.  Correlation analysis on alpha attenuation and nasal skin temperature , 2009 .

[50]  Wang Weiji Research on Empirical Mode Decomposition Denoising Under Multiwavelet Packet , 2015 .

[51]  Ling Guan,et al.  Human emotional state recognition using real 3D visual features from Gabor library , 2013, Pattern Recognit..

[52]  Vincent G. Duffy,et al.  Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement , 2007 .

[53]  N. Kawakami,et al.  Effects of job strain on helper-inducer (CD4+CD29+) and suppressor-inducer (CD4+CD45RA+) T cells in Japanese blue-collar workers. , 1997, Psychotherapy and psychosomatics.

[54]  S. Porges The polyvagal theory: phylogenetic substrates of a social nervous system. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  Nadia Magnenat-Thalmann,et al.  A Highly Efficient Compression Framework for Time-Varying 3-D Facial Expressions , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  Ling Guan,et al.  A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[57]  Mohammed Bennamoun,et al.  An Automatic Framework for Textured 3D Video-Based Facial Expression Recognition , 2014, IEEE Transactions on Affective Computing.

[58]  Dvijesh Shastri,et al.  Imaging Facial Signs of Neurophysiological Responses , 2009, IEEE Transactions on Biomedical Engineering.

[59]  S. Dickerson,et al.  Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. , 2004, Psychological bulletin.

[60]  John H. L. Hansen,et al.  Detection of speech under physical stress: model development, sensor selection, and feature fusion , 2008, INTERSPEECH.

[61]  Liang-Tien Chia,et al.  Local features are not lonely – Laplacian sparse coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[62]  Marc Garbey,et al.  Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery , 2007, IEEE Transactions on Biomedical Engineering.

[63]  John A Updegraff,et al.  Biobehavioral responses to stress in females: tend-and-befriend, not fight-or-flight. , 2000, Psychological review.

[64]  Fanglin Chen,et al.  Including Signal Intensity Increases the Performance of Blind Source Separation on Brain Imaging Data , 2015, IEEE Transactions on Medical Imaging.

[65]  Haibo Li,et al.  Sparse Kernel Reduced-Rank Regression for Bimodal Emotion Recognition From Facial Expression and Speech , 2016, IEEE Transactions on Multimedia.

[66]  Maja Pantic,et al.  Discriminative Shared Gaussian Processes for Multiview and View-Invariant Facial Expression Recognition , 2015, IEEE Transactions on Image Processing.

[67]  Sungyoung Lee,et al.  Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields , 2015, IEEE Transactions on Image Processing.

[68]  Marc T. P. Adam,et al.  Technical Correspondence Blended Emotion Detection for Decision Support , 2015 .

[69]  Ioannis Pavlidis,et al.  The face of fear , 2001 .

[70]  J. Mason A historical view of the stress field. , 1975, Journal of human stress.

[71]  J. Hietanen,et al.  Bodily maps of emotions , 2013, Proceedings of the National Academy of Sciences.

[72]  Marc Moonen,et al.  Distributed Canonical Correlation Analysis in Wireless Sensor Networks With Application to Distributed Blind Source Separation , 2015, IEEE Transactions on Signal Processing.

[73]  Nikolaos G. Bourbakis Detecting Facial Expressions for Monitoring Patterns of Emotional Behavior , 2013, Int. J. Monit. Surveillance Technol. Res..

[74]  Flavio Prieto,et al.  Detection of the Tiredness Level of Drivers Using Machine Vision Techniques , 2011, 2011 IEEE Electronics, Robotics and Automotive Mechanics Conference.

[75]  Oscar Mayora-Ibarra,et al.  Stress modelling and prediction in presence of scarce data , 2016, J. Biomed. Informatics.

[76]  Jia Chen,et al.  Online Distributed Sparsity-Aware Canonical Correlation Analysis , 2016, IEEE Transactions on Signal Processing.

[77]  Chung-Hsien Wu,et al.  Speaking Effect Removal on Emotion Recognition From Facial Expressions Based on Eigenface Conversion , 2013, IEEE Transactions on Multimedia.

[78]  Rosemarie Velik,et al.  Impact of Mental and Physical Stress on Blood Pressure and Pulse Pressure under Normobaric versus Hypoxic Conditions , 2014, PloS one.

[79]  Kunal Mankodiya,et al.  Emotional reactivity monitoring using electrodermal activity analysis in individuals with suicidal behaviors , 2016, 2016 10th International Symposium on Medical Information and Communication Technology (ISMICT).

[80]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[81]  D. Tschoepe,et al.  Effect of mental and physical stress on platelet activation markers in depressed patients and healthy subjects: a pilot study , 2004, Psychiatry Research.

[82]  Paul Landsbergis,et al.  The Workplace And Cardiovascular Disease , 2000 .

[83]  N. Huang,et al.  The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis , 1998, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[84]  Ioannis T. Pavlidis,et al.  StressCam: non-contact measurement of users' emotional states through thermal imaging , 2005, CHI Extended Abstracts.

[85]  Daniël Lakens,et al.  Using a Smartphone to Measure Heart Rate Changes during Relived Happiness and Anger , 2013, IEEE Transactions on Affective Computing.

[86]  Tom Chau,et al.  Classifying Affective States Using Thermal Infrared Imaging of the Human Face , 2010, IEEE Transactions on Biomedical Engineering.

[87]  Mahavir Senan,et al.  Relationship between emotional stress and cardiovascular events. , 2014, Georgian medical news.

[88]  Nicu Sebe,et al.  Learning Personalized Models for Facial Expression Analysis and Gesture Recognition , 2016, IEEE Transactions on Multimedia.

[89]  А. И. Гордиенко,et al.  Влияние хронического психоэмоционального напряжения и уровня физической активности на состояние вазодилатационного резерва у пациентов с артериальной гипертензией и ожирением , 2016 .

[90]  Shin'ichi Satoh,et al.  Estimation of Attentiveness of People Watching TV Based on Their Emotional Behaviors , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[91]  Arcangelo Merla,et al.  Exploring the Use of Thermal Infrared Imaging in Human Stress Research , 2014, PloS one.

[92]  Sheng Hong,et al.  Real-time stress assessment using thermal imaging , 2016, The Visual Computer.

[93]  Changsong Liu,et al.  Sufficient Canonical Correlation Analysis , 2016, IEEE Transactions on Image Processing.

[94]  A. Berstad,et al.  The influence of prolonged physical stress on gastric juice components in healthy man. , 1988, Scandinavian journal of gastroenterology.

[95]  Marc Garbey,et al.  Interacting with human physiology , 2007, Comput. Vis. Image Underst..

[96]  Marie Tahon,et al.  Towards a Small Set of Robust Acoustic Features for Emotion Recognition: Challenges , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[97]  Ying Gao,et al.  Affective Assessment by Digital Processing of the Pupil Diameter , 2013, IEEE Transactions on Affective Computing.

[98]  Nikos Fakotakis,et al.  Modeling the Temporal Evolution of Acoustic Parameters for Speech Emotion Recognition , 2012, IEEE Transactions on Affective Computing.

[99]  Ioannis Pavlidis Continuous physiological monitoring , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[100]  Oscar Mayora-Ibarra,et al.  Automatic Stress Detection in Working Environments From Smartphones’ Accelerometer Data: A First Step , 2015, IEEE Journal of Biomedical and Health Informatics.