A Framework for Emotion Recognition from Human Computer Interaction in Natural Setting

Since human's emotions play a central role in everyday decisions and well-being, developing systems for recognizing and managing human's emotions captured significant research interest in the last decade. However, there is limited research on studying emotion recognition from human-computer interaction (HCI) in natural settings. This work aims at providing a comprehensive study of emotion recognition from HCI, while addressing several remaining challenges in the context of HCI systems. The first challenge incudes the development of HCI emotion recognition models in natural settings instead of lab-controlled settings. The second challenge is to provide a comprehensive collection of potential humans’ interactions with their computers. The third challenge is to provide a meaningful mapping from digital interactions to human related activities, where the mapped activities can then be used as a feature set for accurate emotion recognition models. Hence, the objective of this work is to develop a framework to address these challenges. A robust ground-truth system is defined for the natural capture of a person’s emotion in the context of computer usage while having unobtrusive and seamless data collection. A ground-truth model is designed for emotion recognition by combining facial expressions analysis and selfassessment. New rules are then defined for capturing the digital activity, and then mapping it to human activity that reflects the person’s context and behavior. Finally, the inferred features are used to derive personalized machine learning models for emotion recognition from digital activity. This work also includes a study from real life experiments, where participants were conducting their activity in their natural settings. The inferred features were annotated using the proposed class labels extraction strategy. Finally, a Bayesian Network was used for the emotion recognition model. Results show evidence that it is indeed feasible to sense the user’s emotions through implicit monitoring of everyday computer interactions.

[1]  Maria Virvou,et al.  Emotional Intelligence: Constructing User Stereotypes for Affective Bi-modal Interaction , 2006, KES.

[2]  B. Friedman.  Feelings and the body: The Jamesian perspective on autonomic specificity of emotion , 2010, Biological Psychology.

[3]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[4]  Maria Virvou,et al.  On assisting a visual-facial affect recognition system with keyboard-stroke pattern information , 2010, Knowl. Based Syst..

[5]  Hazem M. Hajj,et al.  A survey of ground-truth in emotion data annotation , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications Workshops.

[6]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Guillaume Chanel,et al.  Emotion Assessment: Arousal Evaluation Using EEG's and Peripheral Physiological Signals , 2006, MRCS.

[8]  Robert C. Holte,et al.  C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats Over-Sampling , 2003 .

[9]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[10]  Egon L. van den Broek,et al.  Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals , 2009, BIOSTEC.

[11]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[12]  Charalampos Bratsas,et al.  On the Classification of Emotional Biosignals Evoked While Viewing Affective Pictures: An Integrated Data-Mining-Based Approach for Healthcare Applications , 2010, IEEE Transactions on Information Technology in Biomedicine.

[13]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[15]  Alex Pentland,et al.  Human computing and machine understanding of human behavior: a survey , 2006, ICMI '06.

[16]  Thomas Jürgensohn,et al.  Comparing Two Emotion Models for Deriving Affective States from Physiological Data , 2008, Affect and Emotion in Human-Computer Interaction.

[17]  Mitsuru Ishizuka,et al.  Affect Analysis Model: novel rule-based approach to affect sensing from text , 2010, Natural Language Engineering.

[18]  Gregory D. Abowd,et al.  A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications , 2001, Hum. Comput. Interact..

[19]  Cecilia Mascolo,et al.  Contextual dissonance: design bias in sensor-based experience sampling methods , 2013, UbiComp.

[20]  E. Kensinger,et al.  Remembering Emotional Experiences: The Contribution of Valence and Arousal , 2004, Reviews in the neurosciences.

[21]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[22]  Cecilia Mascolo,et al.  Smartphones for Large-Scale Behavior Change Interventions , 2013, IEEE Pervasive Computing.

[23]  F. Wilhelm,et al.  Emotions beyond the laboratory: Theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment , 2010, Biological Psychology.

[24]  Cecilia Mascolo,et al.  Opportunities for smartphones in clinical care: the future of mobile mood monitoring. , 2016, The Journal of clinical psychiatry.

[25]  David A. van Leeuwen,et al.  Unobtrusive Multimodal Emotion Detection in Adaptive Interfaces: Speech and Facial Expressions , 2007, HCI.

[26]  Björn W. Schuller,et al.  Emotion representation, analysis and synthesis in continuous space: A survey , 2011, Face and Gesture 2011.

[27]  E Diener.  Introduction to the special section on the structure of emotion. , 1999, Journal of personality and social psychology.

[28]  Sylvia D. Kreibig,et al.  An affective computing approach to physiological emotion specificity: toward subject-independent and stimulus-independent classification of film-induced emotions. , 2011, Psychophysiology.

[29]  Robert M. Hierons,et al.  Towards a Computer Interaction-Based Mood Measure Instrument , 2008, PPIG.

[30]  R. Shah,et al.  Robust model for human activity inference using Bayesian Classifier , 2009 .

[31]  Regan L. Mandryk,et al.  Identifying emotional states using keystroke dynamics , 2011, CHI.

[32]  Thomas Beauvisage,et al.  Computer usage in daily life , 2009, CHI.

[33]  Andreas Ernst,et al.  Face Detection with the Sophisticated High-speed Object Recognition Engine (SHORE) , 2011 .

[34]  Gary Bradski,et al.  Learning-Based Computer Vision with Intels Open Source Computer Vision Library , 2005 .

[35]  Maja Pantic,et al.  The SEMAINE corpus of emotionally coloured character interactions , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[36]  Katarzyna Wac,et al.  Ambulatory Assessment of Affect: Survey of Sensor Systems for Monitoring of Autonomic Nervous Systems Activation in Emotion , 2014, IEEE Transactions on Affective Computing.

[37]  Ralph Gross,et al.  Individual differences in facial expression: stability over time, relation to self-reported emotion, and ability to inform person identification , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[38]  J. Russell.  A circumplex model of affect. , 1980 .

[39]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[40]  Sung-Bae Cho,et al.  A Mobile Context Sharing System Using Activity and Emotion Recognition with Bayesian Networks , 2010, 2010 7th International Conference on Ubiquitous Intelligence & Computing and 7th International Conference on Autonomic & Trusted Computing.

[41]  Hosub Lee,et al.  Towards unobtrusive emotion recognition for affective social communication , 2012, 2012 IEEE Consumer Communications and Networking Conference (CCNC).

[42]  Hang-Bong Kang,et al.  Affective content detection using HMMs , 2003, ACM Multimedia.

[43]  Efthymios Alepis,et al.  Requirements Analysis and Design of an Affective Bi-Modal Intelligent Tutoring System: The Case of Keyboard and Microphone , 2008 .

[44]  Jennifer Healey,et al.  Out of the Lab and into the Fray: Towards Modeling Emotion in Everyday Life , 2010, Pervasive.

[45]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.