Emotion recognition from physiological signals using wireless sensors for presence technologies

In this article we describe a new approach to enhance presence technologies. First, we discuss the strong relationship between cognitive processes and emotions and how human physiology is uniquely affected when experiencing each emotion. Secondly, we introduce our prototype multimodal affective user interface. In the remainder of the paper we describe the emotion elicitation experiment we designed and conducted and the algorithms we implemented to analyse the physiological signals associated with emotions. These algorithms can then be used to recognise the affective states of users from physiological data collected via non-invasive technologies. The affective intelligent user interfaces we plan to create will adapt to user affect dynamically in the current context, thus providing enhanced social presence.

[1]  Christine L. Lisetti,et al.  MAUI: a multimodal affective user interface , 2002, MULTIMEDIA '02.

[2]  Edwin Blake,et al.  The effects of avatars on co-presence in a collaborative virtual environment , 2001 .

[3]  Matthew Lombard,et al.  At the Heart of It All: The Concept of Presence , 2006 .

[4]  James R. Warren,et al.  Supporting special-purpose health care models via adaptive interfaces to the web , 2002, Interact. Comput..

[5]  J. Gross,et al.  Hiding feelings: the acute effects of inhibiting negative and positive emotion. , 1997, Journal of abnormal psychology.

[6]  Christine L. Lisetti,et al.  Modeling Multimodal Expression of User’s Affective Subjective Experience , 2002, User Modeling and User-Adapted Interaction.

[7]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[8]  D. Tucker,et al.  Neural mechanisms of emotion. , 1992, Journal of consulting and clinical psychology.

[9]  A. Manstead,et al.  Handbook of social psychophysiology , 1989 .

[10]  Cristina Conati,et al.  Probabilistic assessment of user's emotions in educational games , 2002, Appl. Artif. Intell..

[11]  Jennifer Healey,et al.  SmartCar: detecting driver stress , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[12]  José-Luis Pérez-de-la-Cruz,et al.  A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation , 2002, User Modeling and User-Adapted Interaction.

[13]  Ted Selker,et al.  COACH: a teaching agent that learns , 1994, CACM.

[14]  Verne E. Lewis,et al.  Mood-congruent vs. mood-state-dependent learning: Implications for a view of emotion. , 1989 .

[15]  Robert Hubal,et al.  Extracting Emotional Information from the Text of Spoken Dialog , 2003 .

[16]  J. Colquitt,et al.  Toward an integrative theory of training motivation: a meta-analytic path analysis of 20 years of research. , 2000, The Journal of applied psychology.

[17]  P. Ekman,et al.  Unmasking the Face: A Guide to Recognizing Emotions From Facial Expressions , 1975 .

[18]  James S. Simkin.  Kinesics and Context: Essays on Body Motion Communication , 1972 .

[19]  R. Zajonc.  On the primacy of affect. , 1984 .

[20]  W. IJsselsteijn,et al.  Elements of a multi-level theory of presence: phenomenology, mental processing and neural correlates , 2002 .

[21]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[22]  Roy S. Kalawsky,et al.  The Validity of Presence as a Reliable Human Performance Metric in Immersive Environments , 2000 .

[23]  Joseph J. Martocchio,et al.  Effects of conceptions of ability on anxiety, self-efficacy, and learning in training. , 1994, The Journal of applied psychology.

[24]  Joseph J. Martocchio,et al.  Relationship between conscientiousness and learning in employee training: mediating influences of self-deception and self-efficacy. , 1997, The Journal of applied psychology.

[25]  Akikazu Takeuchi,et al.  Communicative facial displays as a new conversational modality , 1993, INTERCHI.

[26]  Cynthia LeRouge,et al.  Developing multimodal intelligent affective interfaces for tele-home health care , 2003, Int. J. Hum. Comput. Stud..

[27]  Alfred Kobsa,et al.  A Review and Analysis of Commercial User Modeling Servers for Personalization on the World Wide Web , 2000, User Modeling and User-Adapted Interaction.

[28]  Albert T. Corbett,et al.  Modeling Student Knowledge: Cognitive Tutors in High School and College , 2000, User Modeling and User-Adapted Interaction.

[29]  Michael D. McNeese,et al.  Assessment of User Affective and Belief States for Interface Adaptation: Application to an Air Force Pilot Task , 2002, User Modeling and User-Adapted Interaction.

[30]  Pamela Briggs,et al.  Modelling self-confidence in users of a computer-based system showing unrepresentative design , 1998, Int. J. Hum. Comput. Stud..

[31]  Stephen B. Kaufman,et al.  Home Telemedicine: A Home Health Care Agency Strategy for Maximizing Resources , 1996 .

[32]  Amy L. Baylor,et al.  Beyond Butlers: Intelligent Agents as Mentors , 2000 .

[33]  A. Allen,et al.  Home health visits using a cable television network: user satisfaction. , 1996, Journal of telemedicine and telecare.

[34]  James H. Dulebohn,et al.  PERFORMANCE FEEDBACK EFFECTS IN TRAINING: THE ROLE OF PERCEIVED CONTROLLABILITY , 1994 .

[35]  W. Gardner,et al.  Cognitive, motivation, and affective processes associated with computer-related performance: a path analysis , 2000 .

[36]  P. Warr,et al.  TRAINEE CHARACTERISTICS AND THE OUTCOMES OF OPEN LEARNING , 1995 .

[37]  Jonathan Klein,et al.  This computer responds to user frustration: Theory, design, and results , 2002, Interact. Comput..

[38]  Diane Nahl,et al.  Road Rage and Aggressive Driving: Steering Clear of Highway Warfare , 2000 .

[39]  L. F. Barrett,et al.  Knowing what you're feeling and knowing what to do about it: Mapping the relation between emotion differentiation and emotion regulation , 2001 .

[40]  Nicole Chovil.  Discourse‐oriented facial displays in conversation , 1991 .

[41]  Sara Jones,et al.  The Use of a Co-operative Student Model of Learner Characteristics to Configure a Multimedia Application , 2002, User Modeling and User-Adapted Interaction.

[42]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[43]  Michael J. Pazzani,et al.  User Modeling for Adaptive News Access , 2000, User Modeling and User-Adapted Interaction.

[44]  A. Darkins,et al.  Telemedicine and Telehealth: Principles, Policies, Performance and Pitfalls , 2000 .

[45]  Joseph E LeDoux.  Brain mechanisms of emotion and emotional learning , 1992, Current Opinion in Neurobiology.

[46]  R. Lorenz,et al.  Utility of a Brief Self-Efficacy Scale in Clinical Training Program Evaluation , 2000, Evaluation & the health professions.

[47]  C. Collet,et al.  Autonomic nervous system response patterns specificity to basic emotions. , 1997, Journal of the Autonomic Nervous System.

[48]  G. Bower.  Mood and memory. , 1981, The American psychologist.

[49]  I. Warner.  Telemedicine applications for home health care , 1997, Journal of telemedicine and telecare.

[50]  Penny M. Pexman,et al.  Presenting Your Findings: A Practical Guide for Creating Tables , 1999 .

[51]  Wijnand A. IJsselsteijn,et al.  Presence: concept, determinants, and measurement , 2000, Electronic Imaging.

[52]  J. Freeman,et al.  Aroused and Immersed: The Psychophysiology of Presence , 2000 .

[53]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..