Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals

We discuss the strong relationship between affect and cognition and the importance of emotions in multimodal human computer interaction (HCI) and user modeling. We introduce the overall paradigm for our multimodal system that aims at recognizing its users' emotions and at responding to them accordingly depending upon the current context or application. We then describe the design of the emotion elicitation experiment we conducted by collecting, via wearable computers, physiological signals from the autonomic nervous system (galvanic skin response, heart rate, temperature) and mapping them to certain emotions (sadness, anger, fear, surprise, frustration, and amusement). We show the results of three different supervised learning algorithms that categorize these collected signals in terms of emotions, and generalize their learning to recognize emotions from new collections of signals. We finally discuss possible broader impact and potential applications of emotion recognition for multimodal intelligent systems.

[1]  Christine L. Lisetti,et al.  MAUI: a multimodal affective user interface , 2002, MULTIMEDIA '02.

[2]  Jennifer A. Healey,et al.  Wearable and automotive systems for affect recognition from physiology , 2000 .

[3]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[4]  J. Gross,et al.  Hiding feelings: the acute effects of inhibiting negative and positive emotion. , 1997, Journal of abnormal psychology.

[5]  C. Collet,et al.  Autonomic nervous system response patterns specificity to basic emotions. , 1997, Journal of the Autonomic Nervous System.

[6]  J. C. Dill,et al.  Blood pressure responses and incentive appraisals as a function of perceived ability and objective task demand. , 1993, Psychophysiology.

[7]  G. Bower.  Mood and memory. , 1981, The American psychologist.

[8]  Fatma Nasoz,et al.  Emotion Recognition from Physiological Signals for Presence Technologies , 2004 .

[9]  A. Pecchinenda.  The Affective Significance of Skin Conductance Activity During a Difficult Problem-solving Task , 1996 .

[10]  R. Birdwhistell.  Kinesics and Context: Essays on Body Motion Communication , 1971 .

[11]  C A Smith.  Dimensions of appraisal and physiological response in emotion. , 1989, Journal of personality and social psychology.

[12]  L. Goldstein.  The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction , 1992, The Yale Journal of Biology and Medicine.

[13]  P. Warr,et al.  TRAINEE CHARACTERISTICS AND THE OUTCOMES OF OPEN LEARNING , 1995 .

[14]  Nicole Chovil.  Discourse‐oriented facial displays in conversation , 1991 .

[15]  Verne E. Lewis,et al.  Mood-congruent vs. mood-state-dependent learning: Implications for a view of emotion. , 1989 .

[16]  A. Angrilli,et al.  Cardiac responses associated with affective processing of unpleasant film stimuli. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[17]  Diane Nahl,et al.  Road Rage and Aggressive Driving: Steering Clear of Highway Warfare , 2000 .

[18]  Christine L. Lisetti,et al.  Facial Expression Recognition Using a Neural Network , 1998, FLAIRS.

[19]  P. Ekman,et al.  Unmasking the Face: A Guide to Recognizing Emotions From Facial Expressions , 1975 .

[20]  Joseph J. Martocchio,et al.  Effects of conceptions of ability on anxiety, self-efficacy, and learning in training. , 1994, The Journal of applied psychology.

[21]  D. Goleman.  Emotional Intelligence. New York (Bantam) 1995. , 1995 .

[22]  Manfred Clynes,et al.  Sentics: The touch of emotions , 1977 .

[23]  Frada Burstein,et al.  Modelling the Personality of Decision Makers for Active Decision Support , 1997 .

[24]  J. Stainer,et al.  The Emotions , 1922, Nature.

[25]  R. Wright,et al.  Task difficulty, cardiovascular response, and the magnitude of goal valence. , 1986, Journal of personality and social psychology.

[26]  Joseph E. LeDoux,et al.  Emotion and the amygdala. , 1992 .

[27]  G. Stemmler,et al.  The autonomic differentiation of emotions revisited: convergent and discriminant validation. , 1989, Psychophysiology.

[28]  L. F. Barrett,et al.  Knowing what you're feeling and knowing what to do about it: Mapping the relation between emotion differentiation and emotion regulation , 2001 .

[29]  Mika P. Tarvainen,et al.  Analysis of galvanic skin responses with principal components and clustering techniques , 2001, IEEE Transactions on Biomedical Engineering.

[30]  Greg Linden,et al.  Interactive Assessment of User Preference Models: The Automated Travel Assistant , 1997 .

[31]  P. Lang,et al.  Fear imagery and text processing. , 1986, Psychophysiology.

[32]  Christine L. Lisetti,et al.  Modeling Multimodal Expression of User’s Affective Subjective Experience , 2002, User Modeling and User-Adapted Interaction.

[33]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[34]  Christine L. Lisetti,et al.  Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect , 2000 .

[35]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[36]  Brent Auernheimer,et al.  Physiological data feedback for application in distance education , 2001, PUI '01.

[37]  Christine L. Lisetti,et al.  Emotion recognition from physiological signals using wireless sensors for presence technologies , 2004, Cognition, Technology & Work.

[38]  John Sweller,et al.  Levels of Expertise and User-Adapted Formats of Instructional Presentations: A Cognitive Load Approach , 1997 .

[39]  Wendy S. Ark,et al.  The Emotion Mouse , 1999, HCI.

[40]  J. Lanzetta,et al.  Excitatory strength of expressive faces: effects of happy and fear expressions and context on the extinction of a conditioned fear response. , 1986, Journal of personality and social psychology.

[41]  Susan Bull,et al.  See Yourself Write: A Simple Student Model to Make Students Think , 1997 .

[42]  Penny M. Pexman,et al.  Presenting Your Findings: A Practical Guide for Creating Tables , 1999 .

[43]  Rajita Sinha,et al.  Multivariate Response Patterning of Fear and Anger , 1996 .

[44]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[46]  R. Zajonc.  On the primacy of affect. , 1984 .

[47]  D. Tucker,et al.  Neural mechanisms of emotion. , 1992, Journal of consulting and clinical psychology.

[48]  P. Ekman,et al.  Voluntary facial action generates emotion-specific autonomic nervous system activity. , 1990, Psychophysiology.

[49]  Cynthia LeRouge,et al.  Developing multimodal intelligent affective interfaces for tele-home health care , 2003, Int. J. Hum. Comput. Stud..

[50]  K. Scherer,et al.  The Relationship of Emotion to Cognition: A Functional Approach to a Semantic Controversy , 1987 .

[51]  S. Vrana,et al.  The psychophysiology of disgust: differentiating negative emotional contexts with facial EMG. , 1993, Psychophysiology.

[52]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[53]  P. Ekman,et al.  Emotion and autonomic nervous system activity in the Minangkabau of west Sumatra. , 1992, Journal of personality and social psychology.