Going Further in Affective Computing: How Emotion Recognition Can Improve Adaptive User Interaction

This article joins the fields of emotion recognition and human computer interaction. While much work has been done on recognizing emotions, they are hardly used to improve a user’s interaction with a system. Although the fields of affective computing and especially serious games already make use of detected emotions, they tend to provide application and user specific adaptions only on the task level. We present an approach of utilizing recognized emotions to improve the interaction itself, independent of the underlying application at hand. Examining the state of the art in emotion recognition research and based on the architecture of Companion-System, a generic approach for determining the main cause of an emotion within the history of interactions is presented, allowing a specific reaction and adaption. Using such an approach could lead to systems that use emotions to improve not only the outcome of a task but the interaction itself in order to be truly individual and empathic.

[1]  Oudeyer Pierre-Yves,et al.  The production and recognition of emotions in speech: features and algorithms , 2003 .

[2]  Günther Palm,et al.  Classification of Time Series Utilizing Temporal and Decision Fusion , 2001, Multiple Classifier Systems.

[3]  A. Young,et al.  Emotion Perception from Dynamic and Static Body Expressions in Point-Light and Full-Light Displays , 2004, Perception.

[4]  Christine L. Lisetti,et al.  Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect , 2000 .

[5]  Beth Logan,et al.  Mel Frequency Cepstral Coefficients for Music Modeling , 2000, ISMIR.

[6]  Gilbert Cockton,et al.  Design Principles for Interactive Software , 1997, IFIP — The International Federation for Information Processing.

[7]  Friedhelm Schwenker,et al.  A Hidden Markov Model Based Approach for Facial Expression Recognition in Image Sequences , 2010, ANNPR.

[8]  Philippe A. Palanque,et al.  A Visual and Formal Glue between Application and Interaction , 1999, J. Vis. Lang. Comput..

[9]  Gregor Bertrand,et al.  Context Models for Adaptive Dialogs and Multimodal Interaction , 2013, 2013 9th International Conference on Intelligent Environments.

[10]  Friedhelm Schwenker,et al.  Multimodal Emotion Classification in Naturalistic User Behavior , 2011, HCI.

[11]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[12]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[13]  Bin Yang,et al.  Robust Estimation of Voice Quality Parameters Under Realworld Disturbances , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[14]  J. F. Kelley,et al.  An iterative design methodology for user-friendly natural language office information applications , 1984, TOIS.

[15]  Friedhelm Schwenker,et al.  Fusion paradigms in cognitive technical systems for human-computer interaction , 2015, Neurocomputing.

[16]  Elisabeth André,et al.  Simulating Deceptive Cues of Joy in Humanoid Robots , 2014, IVA.

[17]  F. Itakura Line spectrum representation of linear predictor coefficients of speech signals , 1975 .

[18]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[19]  Jonghwa Kim,et al.  Transsituational Individual-Specific Biopsychological Classification of Emotions , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[20]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[21]  Friedhelm Schwenker,et al.  Fuzzy-Input Fuzzy-Output One-Against-All Support Vector Machines , 2007, KES.

[22]  Elisabeth André,et al.  Comparing Feature Sets for Acted and Spontaneous Speech in View of Automatic Emotion Recognition , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[23]  K. Scherer,et al.  The GRID meets the wheel: assessing emotional feeling via self-report , 2013 .

[24]  Edward I. Altman,et al.  Corporate distress diagnosis: Comparisons using linear discriminant analysis and neural networks (the Italian experience) , 1994 .

[25]  Mark Weiser,et al.  The computer for the 21st Century , 1991, IEEE Pervasive Computing.

[26]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[27]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[28]  Frank Honold,et al.  The Automated Interplay of Multimodal Fission and Fusion in Adaptive HCI , 2014, 2014 International Conference on Intelligent Environments.

[29]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[30]  Catherine Pelachaud,et al.  The TARDIS Framework: Intelligent Virtual Agents for Social Coaching in Job Interviews , 2013, Advances in Computer Entertainment.

[31]  I. Jolliffe Principal Component Analysis , 2002 .

[32]  B. Gelder Towards the neurobiology of emotional body language , 2006, Nature Reviews Neuroscience.

[33]  P. Ekman,et al.  The nature of emotion: Fundamental questions. , 1994 .

[34]  C. Darwin The Expression of the Emotions in Man and Animals , .

[35]  Maja Pantic,et al.  Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge , 2013, AVEC@ACM Multimedia.

[36]  Hiroshi Ishiguro,et al.  Evaluating facial displays of emotion for the android robot Geminoid F , 2011, 2011 IEEE Workshop on Affective Computational Intelligence (WACI).

[37]  Peter J. Thomas,et al.  Proceedings of the 1st international symposium on Handheld and Ubiquitous Computing , 1999 .

[38]  Friedhelm Schwenker,et al.  Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification , 2013, Comput. Speech Lang..

[39]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[40]  Björn W. Schuller,et al.  Deep neural networks for acoustic emotion recognition: Raising the benchmarks , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[41]  Masakazu Matsugu,et al.  Subject independent facial expression recognition with robust face detection using a convolutional neural network , 2003, Neural Networks.

[42]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Lori Lamel,et al.  Challenges in real-life emotion annotation and machine learning based detection , 2005, Neural Networks.

[44]  P. Ekman An argument for basic emotions , 1992 .

[45]  Ginevra Castellano,et al.  Recognising Human Emotions from Body Movement and Gesture Dynamics , 2007, ACII.

[46]  Rong Yan,et al.  Cross-domain video concept detection using adaptive svms , 2007, ACM Multimedia.

[47]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[48]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[49]  Björn W. Schuller,et al.  AVEC 2013: the continuous audio/visual emotion and depression recognition challenge , 2013, AVEC@ACM Multimedia.

[50]  Sascha Meudt,et al.  Prosodic, Spectral and Voice Quality Feature Selection Using a Long-Term Stopping Criterion for Audio-Based Emotion Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[51]  Thorsten Joachims,et al.  Transductive Support Vector Machines , 2006, Semi-Supervised Learning.

[52]  Gregor Bertrand,et al.  Companion-Technology: Towards User- and Situation-Adaptive Functionality of Technical Systems , 2014, 2014 International Conference on Intelligent Environments.

[53]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[54]  Mahir Akgun,et al.  The effect of apologetic error messages and mood states on computer users’ self-appraisal of performance , 2010 .

[55]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[56]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[57]  Friedhelm Schwenker,et al.  Multiple Classifier Systems for the Recogonition of Human Emotions , 2010, MCS.

[58]  Ann Blandford,et al.  Four easy pieces for assessing the usability of multimodal interaction: the CARE properties , 1995, INTERACT.

[59]  Frank Honold,et al.  Multimodal Interaction History and its use in Error Detection and Recovery , 2014, ICMI.

[60]  Elaine M. Raybourn,et al.  A new paradigm for serious games: Transmedia learning for more effective training and education , 2014, J. Comput. Sci..

[61]  Frank Honold,et al.  The Nonverbal Toolkit: Towards a Framework for Automatic Integration of Nonverbal Communication into Virtual Environments , 2012, 2012 Eighth International Conference on Intelligent Environments.

[62]  Andreas Wendemuth,et al.  Companion-Technology for Cognitive Technical Systems , 2011, KI - Künstliche Intelligenz.

[63]  Frank Honold,et al.  Adaptive probabilistic fission for multimodal systems , 2012, OZCHI.

[64]  Hani Hagras,et al.  Advanced Intelligent Environments , 2009 .

[65]  Günther Palm,et al.  Combination of sequential class distributions from multiple channels using Markov fusion networks , 2014, Journal on Multimodal User Interfaces.

[66]  W. Wundt Grundriss der Psychologie , 1896 .

[67]  Gregor Bertrand,et al.  GEEDI - Guards for Emotional and Explanatory DIalogues , 2010, 2010 Sixth International Conference on Intelligent Environments.

[68]  Sascha Meudt,et al.  On Instance Selection in Audio Based Emotion Recognition , 2012, ANNPR.

[69]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[70]  Gwen Littlewort,et al.  The computer expression recognition toolbox (CERT) , 2011, Face and Gesture 2011.

[71]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[73]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[74]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[75]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[76]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[77]  Friedhelm Schwenker,et al.  Conditioned Hidden Markov Model Fusion for Multimodal Classification , 2011, INTERSPEECH.

[78]  G. Palm,et al.  Classifier fusion for emotion recognition from speech , 2007 .

[79]  Andreas Wendemuth,et al.  Intraindividual and interindividual multimodal emotion analyses in Human-Machine-Interaction , 2012, 2012 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support.