The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition

This paper reports on a new methodology for the automatic assessment of emotional responses. More specifically, emotions are elicited in agreement with a bidimensional spatial localization of affective states, that is, arousal and valence dimensions. A dedicated experimental protocol was designed and realized where specific affective states are suitably induced while three peripheral physiological signals, i.e., ElectroCardioGram (ECG), ElectroDermal Response (EDR), and ReSPiration activity (RSP), are simultaneously acquired. A group of 35 volunteers was presented with sets of images gathered from the International Affective Picture System (IAPS) having five levels of arousal and five levels of valence, including a neutral reference level in both. Standard methods as well as nonlinear dynamic techniques were used to extract sets of features from the collected signals. The goal of this paper is to implement an automatic multiclass arousal/valence classifier comparing performance when extracted features from nonlinear methods are used as an alternative to standard features. Results show that, when nonlinearly extracted features are used, the percentages of successful recognition dramatically increase. A good recognition accuracy (>;90 percent) after 40-fold cross-validation steps for both arousal and valence classes was achieved by using the Quadratic Discriminant Classifier (QDC).

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  R. Spitzer,et al.  The PHQ-9 , 2001, Journal of General Internal Medicine.

[3]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1956 .

[4]  Shih-Fu Chang,et al.  Blind detection of photomontage using higher order statistics , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[5]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[6]  Christine L. Lisetti,et al.  Can a rational agent afford to be affectless? a formal approach , 2002, Appl. Artif. Intell..

[7]  Georgios N. Yannakakis,et al.  Entertainment modeling through physiology in physical play , 2008, Int. J. Hum. Comput. Stud..

[8]  J. Davitz.  The language of emotion , 1969 .

[9]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[10]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[11]  J. Stainer,et al.  The Emotions , 1922, Nature.

[12]  U. Rajendra Acharya,et al.  Heart rate variability: a review , 2006, Medical and Biological Engineering and Computing.

[13]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[14]  Changchun Liu,et al.  Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder , 2008, Int. J. Hum. Comput. Stud..

[15]  Jerry M. Mendel,et al.  Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications , 1991, Proc. IEEE.

[16]  M. Milanova,et al.  Recognition of Emotional states in Natural Human-Computer Interaction , 2008, 2008 IEEE International Symposium on Signal Processing and Information Technology.

[17]  D. Ruelle,et al.  Recurrence Plots of Dynamical Systems , 1987 .

[18]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[19]  R. Cohen,et al.  An Efficient Algorithm for Spectral Analysis of Heart Rate Variability , 1986, IEEE Transactions on Biomedical Engineering.

[20]  Egon L. van den Broek,et al.  Personalized affective music player , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[21]  Christos D. Katsis,et al.  An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders , 2011, Biomed. Signal Process. Control..

[22]  H. Mccurdy,et al.  Consciousness and the galvanometer. , 1950, Psychological review.

[23]  R. Krauss,et al.  Facial and autonomic manifestations of the dimensional structure of emotion , 1984 .

[24]  P. Ekman.  Universal facial expressions of emotion. , 1970 .

[25]  Jürgen Kurths,et al.  Recurrence plots for the analysis of complex systems , 2009 .

[26]  Nadia Magnenat-Thalmann,et al.  A Model for Personality and Emotion Simulation , 2003, KES.

[27]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[28]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  R. Plutchik.  Emotions : a general psychoevolutionary theory , 1984 .

[30]  Woontack Woo,et al.  Physiological Sensing and Feature Extraction for Emotion Recognition by Exploiting Acupuncture Spots , 2005, ACII.

[31]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[33]  Gerald Young,et al.  Approaches to Emotions , 2011 .

[34]  C. M. Lim,et al.  Cardiac state diagnosis using higher order spectra of heart rate variability , 2008, Journal of medical engineering & technology.

[35]  Sun Kook Yoo,et al.  Neural Network Based Emotion Estimation Using Heart Rate Variability and Skin Resistance , 2005, ICNC.

[36]  Bruce Bowe.  The Face of Emotion , 1985 .

[37]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[38]  J. Westerink,et al.  Considerations for emotion-aware consumer products. , 2009, Applied ergonomics.

[39]  Jennifer A. Healey,et al.  Affect detection in the real world: Recording and processing physiological signals , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[40]  C. Goose,et al.  Glossary of Terms , 2004, Machine Learning.

[41]  Mohammad Soleymani,et al.  Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos , 2010, Brain Informatics.

[42]  Peter Boesiger,et al.  Segregated neural representation of distinct emotion dimensions in the prefrontal cortex—an fMRI study , 2006, NeuroImage.

[43]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[44]  Vasilis Z. Marmarelis,et al.  Nonlinear Dynamic Modeling of Physiological Systems: Marmarelis/Nonlinear , 2004 .

[45]  K. Poels,et al.  How to Capture the Heart? Reviewing 20 Years of Emotion Measurement in Advertising , 2006 .

[46]  H. Akaike.  Fitting autoregressive models for prediction , 1969 .

[47]  Vasilis Z. Marmarelis,et al.  Nonlinear Dynamic Modeling of Physiological Systems , 2004 .

[48]  J. Russell.  A circumplex model of affect. , 1980 .

[49]  H. Schlosberg.  Three dimensions of emotion. , 1954, Psychological review.

[50]  D. Weinberger,et al.  Neocortical modulation of the amygdala response to fearful stimuli , 2003, Biological Psychiatry.

[51]  C. L. Nikias,et al.  Higher-order spectra analysis : a nonlinear signal processing framework , 1993 .

[52]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[53]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  A. Barreto,et al.  Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[55]  J. G. Taylor,et al.  Emotion recognition in human-computer interaction , 2005, Neural Networks.

[56]  Changchun Liu,et al.  An empirical study of machine learning techniques for affect recognition in human–robot interaction , 2006, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[57]  S. Tomkins,et al.  Affect Imagery Consciousness: The Positive Affects , 1963 .

[58]  J. Vaidya,et al.  The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence , 1999 .

[59]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[60]  H. Stanley,et al.  Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. , 1995, Chaos.

[61]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[62]  K. Scherer,et al.  Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization , 2008, Consciousness and Cognition.

[63]  Egon L. van den Broek,et al.  Mood Swings: An Affective Interactive Art System , 2009, INTETAIN.

[64]  Norbert Marwan,et al.  Selection of recurrence threshold for signal detection , 2008 .

[65]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[66]  D. Watson,et al.  On traits and temperament: general and specific factors of emotional experience and their relation to the five-factor model. , 1992, Journal of personality.

[67]  P. Lang.  International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .

[68]  J. Lagopoulos.  Electrodermal activity , 2007, Acta Neuropsychiatrica.

[69]  P. Lang.  Behavioral treatment and bio-behavioral assessment: computer applications , 1980 .

[70]  M. Bradley,et al.  Emotion, Motivation, and Anxiety: Brain Mechanisms and Psychophysiology the Motivational Organization of Emotion Patterns of Human Emotion Emotion and Perception the Psychophysiology of Picture Processing Neural Imaging: Motivation in the Visual Cortex Motivational Circuits in the Brain , 2022 .

[71]  R. Orglmeister,et al.  The principles of software QRS detection , 2002, IEEE Engineering in Medicine and Biology Magazine.

[72]  J. M. Carroll,et al.  On the bipolarity of positive and negative affect. , 1999, Psychological bulletin.

[73]  T. Dalgleish.  Basic Emotions , 2004 .

[74]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[75]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[76]  Willis J. Tompkins,et al.  A Real-Time QRS Detection Algorithm , 1985, IEEE Transactions on Biomedical Engineering.

[77]  F. Schlindwein,et al.  A study on the optimum order of autoregressive models for heart rate variability. , 2002, Physiological measurement.

[78]  A. N. Ishchenko,et al.  Automated complex for multiparameter analysis of the galvanic skin response signal , 1989 .

[79]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[80]  Enzo Pasquale Scilingo,et al.  Comparative Evaluation of Susceptibility to Motion Artifact in Different Wearable Systems for Monitoring Respiratory Rate , 2010, IEEE Transactions on Information Technology in Biomedicine.

[81]  J. Panksepp.  Affective Neuroscience: The Foundations of Human and Animal Emotions , 1998 .

[82]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[83]  Egon L. van den Broek,et al.  Unobtrusive Sensing of Emotions (USE) , 2009, J. Ambient Intell. Smart Environ..

[84]  J. M. Kittross.  The measurement of meaning , 1959 .

[85]  M Tulppo,et al.  RSA component extraction from heart rate signal by independent component analysis , 2009, 2009 36th Annual Computers in Cardiology Conference (CinC).

[86]  M. Kreutzer,et al.  DARWIN and FACIAL EXPRESSION A CENTURY OF RESEARCH IN REVIEW , 2014 .

[87]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[88]  A. Damasio,et al.  Basic emotions are associated with distinct patterns of cardiorespiratory activity. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[89]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[90]  T. Subba Rao,et al.  Classification, Parameter Estimation and State Estimation: An Engineering Approach Using MATLAB , 2004 .

[91]  Klaus R Scherer,et al.  Emotions are emergent processes: they require a dynamic computational architecture , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[92]  Graham Clarke,et al.  A user-independent real-time emotion recognition system for software agents in domestic environments , 2007, Eng. Appl. Artif. Intell..