Emotions and Mood States: Modeling, Elicitation, and Classification

This chapter reports on the current state of the art related to the study of autonomic nervous system (ANS) dynamics for mood and emotional-state assessment. Accordingly, several methodological approaches and guidelines for the development of an automatic system for an effective characterization and recognition of emotions are described along.

[1]  H. Stampfer,et al.  The Relationship between Psychiatric Illness and the Circadian Pattern of Heart Rate , 1998, The Australian and New Zealand journal of psychiatry.

[2]  Kevin S. LaBar,et al.  Mood Alters Amygdala Activation to Sad Distractors During an Attentional Task , 2006, Biological Psychiatry.

[3]  J. Hirsch,et al.  Respiratory sinus arrhythmia in humans: how breathing pattern modulates heart rate. , 1981, The American journal of physiology.

[4]  Peter J. Lang,et al.  Gaze Patterns When Looking at Emotional Pictures: Motivationally Biased Attention , 2004 .

[5]  K. Poels,et al.  How to Capture the Heart? Reviewing 20 Years of Emotion Measurement in Advertising , 2006 .

[6]  P. Ekman,et al.  Approaches To Emotion , 1985 .

[7]  F. Mormann,et al.  Mean phase coherence as a measure for phase synchronization and its application to the EEG of epilepsy patients , 2000 .

[8]  M. Bradley,et al.  Emotion, Motivation, and Anxiety: Brain Mechanisms and Psychophysiology the Motivational Organization of Emotion Patterns of Human Emotion Emotion and Perception the Psychophysiology of Picture Processing Neural Imaging: Motivation in the Visual Cortex Motivational Circuits in the Brain , 2022 .

[9]  J. Beatty,et al.  The pupillary system. , 2000 .

[10]  Jeffrey C. Smith,et al.  Neuronal pacemaker for breathing visualized in vitro , 1999, Nature.

[11]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[12]  Sheri L. Johnson,et al.  Emotion and Bipolar Disorder. , 2007 .

[13]  Georgios N. Yannakakis,et al.  Entertainment modeling through physiology in physical play , 2008, Int. J. Hum. Comput. Stud..

[14]  Peter Boesiger,et al.  Segregated neural representation of distinct emotion dimensions in the prefrontal cortex—an fMRI study , 2006, NeuroImage.

[15]  Jeffrey M. Hausdorff,et al.  Fractal mechanisms and heart rate dynamics. Long-range correlations and their breakdown with disease. , 1995, Journal of electrocardiology.

[16]  Makoto Fukumoto,et al.  Effects of musical tempo on multiple subjective impressions , 2010, Int. J. Biom..

[17]  Jürgen Kurths,et al.  Detection of n:m Phase Locking from Noisy Data: Application to Magnetoencephalography , 1998 .

[18]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[19]  Lakhmi C. Jain,et al.  Knowledge-Based Intelligent Information and Engineering Systems , 2004, Lecture Notes in Computer Science.

[20]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[21]  R Quian Quiroga,et al.  Performance of different synchronization measures in real data: a case study on electroencephalographic signals. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[22]  J. Kurths,et al.  Phase synchronization in noisy and chaotic oscillators , 1997 .

[23]  Michael T. Ghiselin,et al.  Darwin and Facial Expression: A Century of Research in Review.@@@Darwin on Man: A Psychological Study of Scientific Creativity. , 1974 .

[24]  Emery N. Brown,et al.  Dynamic Assessment of Baroreflex Control of Heart Rate During Induction of Propofol Anesthesia Using a Point Process Method , 2010, Annals of Biomedical Engineering.

[25]  H. Schlosberg.  Three dimensions of emotion. , 1954, Psychological review.

[26]  Stefano Pini,et al.  Prevalence and burden of bipolar disorders in European countries , 2005, European Neuropsychopharmacology.

[27]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[28]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[29]  Gregory Levitin,et al.  Algorithm for Reliability Evaluation of Nonrepairable Phased-Mission Systems Consisting of Gradually Deteriorating Multistate Elements , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[30]  J. Davitz.  The language of emotion , 1969 .

[31]  Gerald L. Lohse,et al.  Organizational Behavior and Human Decision Processes a Comparison of Two Process Tracing Methods for Choice Tasks , 2022 .

[32]  M. Bradley,et al.  Activation of the visual cortex in motivated attention. , 2003, Behavioral neuroscience.

[33]  S. Havlin,et al.  Experimental evidence for phase synchronization transitions in the human cardiorespiratory system. , 2007, Physical review letters.

[34]  J. Richman,et al.  Physiological time-series analysis using approximate entropy and sample entropy. , 2000, American journal of physiology. Heart and circulatory physiology.

[35]  R. Levine,et al.  Culture and personality : contemporary readings , 1976 .

[36]  Natalia M. Arzeno,et al.  Chaotic Signatures of Heart Rate Variability and Its Power Spectrum in Health, Aging and Heart Failure , 2009, PloS one.

[37]  J. Vaidya,et al.  The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence , 1999 .

[38]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[39]  Guillaume Chanel,et al.  Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[40]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  M. Moser,et al.  Wirkungen der Therapeutischen Sprachgestaltung auf Herzfrequenz-Variabilität und Befinden , 2001, Complementary Medicine Research.

[42]  P Lemoine,et al.  Heart rate circadian rhythm as a biological marker of desynchronization in major depression: a methodological and preliminary report. , 1990, Chronobiology international.

[43]  Veikko Surakka,et al.  Pupil size variation as an indication of affective processing , 2003, Int. J. Hum. Comput. Stud..

[44]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[45]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[46]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[47]  J. Kurths,et al.  Attractor-Repeller Collision and Eyelet Intermittency at the Transition to Phase Synchronization , 1997 .

[48]  K. Sunagawa,et al.  Dynamic nonlinear vago-sympathetic interaction in regulating heart rate , 2005, Heart and Vessels.

[49]  Sheri L. Johnson,et al.  Emotion and psychopathology: Bridging affective and clinical science. , 2007 .

[50]  P. Lang.  International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .

[51]  P Lemoine,et al.  Sleep and heart rate circadian rhythm in depression: the necessity to separate. , 1993, Chronobiology international.

[52]  H. Mccurdy,et al.  Consciousness and the galvanometer. , 1950, Psychological review.

[53]  H. Herzel,et al.  Analyzing entrainment of heartbeat and respiration with surrogates , 1998, IEEE Engineering in Medicine and Biology Magazine.

[54]  A. Damasio,et al.  Basic emotions are associated with distinct patterns of cardiorespiratory activity. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[55]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[56]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  C. Peng,et al.  What is physiologic complexity and how does it change with aging and disease? , 2002, Neurobiology of Aging.

[58]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[59]  D. Watson,et al.  On traits and temperament: general and specific factors of emotional experience and their relation to the five-factor model. , 1992, Journal of personality.

[60]  Enrico Smeraldi,et al.  Neural responses to emotional stimuli in comorbid borderline personality disorder and bipolar depression , 2012, Psychiatry Research: Neuroimaging.

[61]  David B. Kaber,et al.  Emotional State Classification in Patient–Robot Interaction Using Wavelet Analysis and Statistics-Based Feature Selection , 2013, IEEE Transactions on Human-Machine Systems.

[62]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[63]  Lan Li,et al.  Emotion Recognition Using Physiological Signals from Multiple Subjects , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[64]  Gerald Young,et al.  Approaches to Emotions , 2011 .

[65]  Enzo Pasquale Scilingo,et al.  Oscillations of Heart Rate and Respiration Synchronize During Affective Visual Stimulation , 2012, IEEE Transactions on Information Technology in Biomedicine.

[66]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[67]  Changchun Liu,et al.  An empirical study of machine learning techniques for affect recognition in human–robot interaction , 2006, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  T. Musha,et al.  1/f Fluctuation of Heartbeat Period , 1982, IEEE Transactions on Biomedical Engineering.

[69]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[70]  J. Kurths,et al.  Heartbeat synchronized with ventilation , 1998, Nature.

[71]  I. Christie,et al.  Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[72]  Jennifer A. Healey,et al.  Affect detection in the real world: Recording and processing physiological signals , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[73]  A. Glassman,et al.  Depression, Cardiac Death, and the Central Nervous System , 1998, Neuropsychobiology.

[74]  P. Gomez,et al.  Affective and physiological responses to environmental noises and music. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[75]  F Atyabi,et al.  Two Statistical Methods for Resolving Healthy Individuals and Those with Congestive Heart Failure Based on Extended Self-similarity and a Recursive Method , 2006, Journal of biological physics.

[76]  R. Krauss,et al.  Facial and autonomic manifestations of the dimensional structure of emotion , 1984 .

[77]  S Akselrod,et al.  Does synchronization reflect a true interaction in the cardiorespiratory system? , 2002, Medical engineering & physics.

[78]  H. Lackner,et al.  Oscillations of heart rate and respiration synchronize during poetry recitation. , 2004, American journal of physiology. Heart and circulatory physiology.

[79]  Raymond E. Ideker,et al.  Chaos in the Heart: Implications for Clinical Cardiology , 1990, Bio/Technology.

[80]  W. Wundt.  Grundriss der Psychologie , 1896 .

[81]  G Sugihara,et al.  Nonlinear control of heart rate variability in human infants. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[82]  Rosalind W. Picard.  Affective Computing , 1997 .

[83]  U. Rajendra Acharya,et al.  Heart rate variability: a review , 2006, Medical and Biological Engineering and Computing.

[84]  R. Plutchik.  Emotions : a general psychoevolutionary theory , 1984 .

[85]  Enzo Pasquale Scilingo,et al.  A novel EDA glove based on textile-integrated electrodes for affective computing , 2012, Medical & Biological Engineering & Computing.

[86]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[87]  Emery N. Brown,et al.  Characterizing nonlinear heartbeat dynamics within a point process framework , 2010, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[88]  A. W. Siegman,et al.  Nonverbal behavior and communication , 1979 .

[89]  J. Russell.  A circumplex model of affect. , 1980 .

[90]  L. Glass,et al.  From Clocks to Chaos: The Rhythms of Life , 1988 .

[91]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[92]  Emery N. Brown,et al.  Analysis of heartbeat dynamics by point process adaptive filtering , 2006, IEEE Transactions on Biomedical Engineering.

[93]  R. Kessler,et al.  Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States. Results from the National Comorbidity Survey. , 1994, Archives of general psychiatry.

[94]  J. Westerink,et al.  Considerations for emotion-aware consumer products. , 2009, Applied ergonomics.

[95]  I Hagerman,et al.  Chaos-related deterministic regulation of heart rate variability in time- and frequency domains: effects of autonomic blockade and exercise. , 1996, Cardiovascular research.

[96]  Graham Clarke,et al.  A user-independent real-time emotion recognition system for software agents in domestic environments , 2007, Eng. Appl. Artif. Intell..

[97]  Christos D. Katsis,et al.  An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders , 2011, Biomed. Signal Process. Control..

[98]  Nadia Magnenat-Thalmann,et al.  A Model for Personality and Emotion Simulation , 2003, KES.

[99]  R. Levenson.  Autonomic Nervous System Differences among Emotions , 1992 .

[100]  D. Cysarz,et al.  Cardiorespiratory synchronization during Zen meditation , 2005, European Journal of Applied Physiology.

[101]  J. Panksepp.  Affective Neuroscience: The Foundations of Human and Animal Emotions , 1998 .

[102]  C. Izard.  The face of emotion , 1971 .

[103]  Carroll,et al.  Synchronization in chaotic systems. , 1990, Physical review letters.

[104]  Yew-Soon Ong,et al.  Advances in Natural Computation, First International Conference, ICNC 2005, Changsha, China, August 27-29, 2005, Proceedings, Part I , 2005, ICNC.

[105]  Mika P. Tarvainen,et al.  Analysis of galvanic skin responses with principal components and clustering techniques , 2001, IEEE Transactions on Biomedical Engineering.

[106]  A Calciati,et al.  Evidence for an intrinsic mechanism regulating heart rate variability in the transplanted and the intact heart during submaximal dynamic exercise? , 1990, Cardiovascular research.

[107]  Martin Hasler,et al.  State Change Detection Using Multivariate Synchronization Measure from Physiological Signals (Special Section on Papers Awarded the Student Paper Award at NCSP'06) , 2006 .

[108]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[109]  H H Abel,et al.  Synchronization in the human cardiorespiratory system. , 1998, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[110]  J. Saul,et al.  Beat-to-Beat Variations of Heart Rate Reflect Modulation of Cardiac Autonomic Outflow , 1990 .

[111]  M. Phillips,et al.  Identifying functional neuroimaging biomarkers of bipolar disorder: toward DSM-V. , 2007, Schizophrenia bulletin.

[112]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[113]  T. A. Williams,et al.  Technology in mental health care delivery systems , 1980 .

[114]  Eduard Vieta,et al.  Staging Bipolar Disorder , 2011, Neurotoxicity Research.

[115]  Laila Dybkjær,et al.  Affective Dialogue Systems , 2004, Lecture Notes in Computer Science.

[116]  M. P. Janisse.  Pupil size, affect and exposure frequency , 1974 .

[117]  Chi-Sang Poon,et al.  Decrease of cardiac chaos in congestive heart failure , 1997, Nature.

[118]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[119]  E. Brown,et al.  A point-process model of human heartbeat intervals: new definitions of heart rate and heart rate variability. , 2005, American journal of physiology. Heart and circulatory physiology.

[120]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[121]  Egon L. van den Broek,et al.  Personalized affective music player , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[122]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[123]  Christine L. Lisetti,et al.  Emotion recognition from physiological signals using wireless sensors for presence technologies , 2004, Cognition, Technology & Work.

[124]  John T. Cacioppo,et al.  Techniques in Psychophysiology , 1980 .

[125]  J. M. Carroll,et al.  On the bipolarity of positive and negative affect. , 1999, Psychological bulletin.

[126]  J. Andreassi.  Psychophysiology: Human Behavior and Physiological Response , 1980 .

[127]  Egon L. van den Broek,et al.  Unobtrusive Sensing of Emotions (USE) , 2009, J. Ambient Intell. Smart Environ..

[128]  Christine L. Lisetti,et al.  Can a rational agent afford to be affectless? a formal approach , 2002, Appl. Artif. Intell..

[129]  T. Seppänen,et al.  Physiological Background of the Loss of Fractal Heart Rate Dynamics , 2005, Circulation.

[130]  M D Prokhorov,et al.  Synchronization between main rhythmic processes in the human cardiovascular system. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[131]  M. Rosenstein,et al.  A practical method for calculating largest Lyapunov exponents from small data sets , 1993 .

[132]  Emery N. Brown,et al.  Assessment of Autonomic Control and Respiratory Sinus Arrhythmia Using Point Process Models of Human Heart Beat Dynamics , 2009, IEEE Transactions on Biomedical Engineering.

[133]  Kurths,et al.  Phase synchronization of chaotic oscillators. , 1996, Physical review letters.

[134]  E. Scilingo,et al.  Dominant Lyapunov exponent and approximate entropy in heart rate variability during emotional visual elicitation , 2012, Front. Neuroeng..

[135]  Rynson W. H. Lau,et al.  Advances in Artificial Reality and Tele-Existence, 16th International Conference on Artificial Reality and Telexistence, ICAT 2006, Hangzhou, China, November 29 - December 1, 2006, Proceedings , 2006, ICAT.

[136]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[137]  Grant L. Iverson,et al.  Reliability of Circadian Heart Pattern Analysis in Psychiatry , 2004, Psychiatric Quarterly.

[138]  D. Weinberger,et al.  Neocortical modulation of the amygdala response to fearful stimuli , 2003, Biological Psychiatry.

[139]  D. Ruelle,et al.  Recurrence Plots of Dynamical Systems , 1987 .

[140]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[141]  Chengqi Zhang,et al.  Propagating temporal relations of intervals by matrix , 2002, Appl. Artif. Intell..

[142]  H. Wittchen,et al.  Size and burden of mental disorders in Europe—a critical review and appraisal of 27 studies , 2005, European Neuropsychopharmacology.

[143]  S. Feldstein,et al.  Pupillary Behavior in Communication , 2014 .

[144]  Changchun Liu,et al.  Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder , 2008, Int. J. Hum. Comput. Stud..

[145]  Robert M Carney,et al.  Association of anxiety with reduced baroreflex cardiac control in patients after acute myocardial infarction. , 2002, American heart journal.

[146]  Enzo Pasquale Scilingo,et al.  The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition , 2012, IEEE Transactions on Affective Computing.

[147]  Amit Konar,et al.  Emotion Recognition From Facial Expressions and Its Control Using Fuzzy Logic , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[148]  Enzo Pasquale Scilingo,et al.  Eye gaze patterns in emotional pictures , 2012, Journal of Ambient Intelligence and Humanized Computing.

[149]  N. A. Coulter,et al.  Respiratory sinus arrhythmia: a frequency dependent phenomenon , 1964 .

[150]  J. Kurths,et al.  Recurrence-plot-based measures of complexity and their application to heart-rate-variability data. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[151]  Woontack Woo,et al.  Physiological Sensing and Feature Extraction for Emotion Recognition by Exploiting Acupuncture Spots , 2005, ACII.

[152]  Xiaolin Yu,et al.  Effects of mental tasks on the cardiorespiratory synchronization , 2010, Respiratory Physiology & Neurobiology.

[153]  Geoffrey R. Loftus,et al.  Eye fixations and memory for emotional events. , 1991 .

[154]  A. Malliani,et al.  Heart rate variability. Standards of measurement, physiological interpretation, and clinical use , 1996 .

[155]  Edward J. Rzempoluck,et al.  A New Potential Marker for Abnormal Cardiac Physiology in Depression , 2005, Journal of Behavioral Medicine.

[156]  Sun Kook Yoo,et al.  Neural Network Based Emotion Estimation Using Heart Rate Variability and Skin Resistance , 2005, ICNC.

[157]  Benicio N Frey,et al.  Oxidative stress markers in bipolar disorder: a meta-analysis. , 2008, Journal of affective disorders.

[158]  Egon L. van den Broek,et al.  Mood Swings: An Affective Interactive Art System , 2009, INTETAIN.

[159]  P. Grassberger,et al.  Measuring the Strangeness of Strange Attractors , 1983 .

[160]  R. Carney,et al.  Depression as a risk factor for cardiac events in established coronary heart disease: A review of possible mechanisms , 1995, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[161]  A. Guyton,et al.  Textbook of Medical Physiology , 1961 .

[162]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[163]  A L Goldberger,et al.  Decreased neuroautonomic complexity in men during an acute major depressive episode: analysis of heart rate dynamics , 2011, Translational Psychiatry.

[164]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[165]  L. Glass.  Synchronization and rhythmic processes in physiology , 2001, Nature.

[166]  D. Jewett,et al.  Activity of single efferent fibres in the cervical vagus nerve of the dog, with special reference to possible cardio‐inhibitory fibres , 1964, The Journal of physiology.

[167]  A. Barreto,et al.  Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.