On the Use of Multi-attribute Decision Making for Combining Audio-Lingual and Visual-Facial Modalities in Emotion Recognition

In this chapter, we present and discuss a novel approach that we have developed for the integration of audio-lingual and visual-facial modalities in a bi-modal user interface for affect recognition. Even though researchers acknowledge that two modalities can provide information that is complementary to each other with respect to affect recognition, satisfactory progress has not yet been achieved towards the integration of the two modalities. In our research reported herein, we approach the combination of the two modalities from the perspective of a human observer by employing a multi-criteria decision making theory for dynamic affect recognition of computer users. Our approach includes the specification of the strengths and weaknesses of each modality with respect to affect recognition concerning the 6 basic emotion states, namely happiness, sadness, surprise, anger and disgust, as well as the emotionless state which we refer to as neutral. We present two empirical studies that we have conducted involving human users and human observers concerning the recognition of emotions from audio-lingual and visual-facial modalities. Based on the results of the empirical studies, we assign weights to criteria for the application of a multi-criteria decision making theory. Additionally, the results of the empirical studies provide information that may be used by other researchers in the field of affect recognition and is currently unavailable in the relevant literature.

[1]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[2]  T. Dalgleish.  Basic Emotions , 2004 .

[3]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[4]  Tsutomu Miyasato,et al.  Bimodal Emotion Recognition by Man and Machine , 2007 .

[5]  Evangelos Triantaphyllou,et al.  An examination of the effectiveness of multi-dimensional decision-making methods: A decision-making paradox , 1989, Decis. Support Syst..

[6]  P. Ekman.  Unmasking The Face , 1975 .

[7]  Aaron Sloman,et al.  Fundamental Questions , 2006, KI.

[8]  P. Johnson-Laird,et al.  Towards a Cognitive Theory of Emotions , 1987 .

[9]  S. Tomkins.  AFFECT AS AMPLIFICATION: SOME MODIFICATIONS IN THEORY , 1980 .

[10]  David N. Chin,et al.  Acquiring User Preferences for Product Customization , 2001, User Modeling.

[11]  Maria Virvou,et al.  On assisting a visual-facial affect recognition system with keyboard-stroke pattern information , 2010, Knowl. Based Syst..

[12]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[13]  Magda B. Arnold,et al.  The nature of emotion , 1968 .

[14]  B. Weiner.  An attributional theory of achievement motivation and emotion. , 1985, Psychological review.

[15]  Maria Virvou,et al.  Object Oriented Design for Multiple Modalities in Affective Interaction , 2014 .

[16]  Pc,et al.  An Introduction to Social Psychology , 1920, Nature.

[17]  Philippe Vincke,et al.  Multicriteria Decision-aid , 1993 .

[18]  Anthony Jameson,et al.  Decision-Theoretic Planning Meets User Requirements: Enhancements and Studies of an Intelligent Shopping Guide , 2005, Pervasive.

[19]  Bruce Bowe.  The Face of Emotion , 1985 .

[20]  Zhihong Zeng,et al.  Audio-Visual Affect Recognition , 2007, IEEE Transactions on Multimedia.

[21]  P. Young,et al.  Emotion and personality , 1963 .

[22]  Daniel Kudenko,et al.  Group Decision Making through Mediated Discussions , 2003, User Modeling.

[23]  Sharon Oviatt,et al.  User-centered modeling and evaluation of multimodal interfaces , 2003, Proc. IEEE.

[24]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[25]  R. Gunderman,et al.  Emotional intelligence. , 2011, Journal of the American College of Radiology : JACR.

[26]  Zhigang Deng,et al.  Analysis of emotion recognition using facial expressions, speech and multimodal information , 2004, ICMI '04.

[27]  Ioanna-Ourania Stathopoulou,et al.  Comparative performance evaluation of classifiers for Facial Expression Recognition , 2009, KES IIMSS.

[28]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[29]  Ralph Schäfer,et al.  Bayesian Networks for Estimating the User ' s Interests in the Context of a Configuration Task , 2001 .

[30]  J. Kagan,et al.  On the nature of emotion. , 1994, Monographs of the Society for Research in Child Development.

[31]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[32]  P. Ekman.  Emotion in the human face , 1982 .

[33]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[34]  Christine L. Lisetti,et al.  MAUI avatars: Mirroring the user's sensed emotions via expressive multi-ethnic facial avatars , 2006, J. Vis. Lang. Comput..

[35]  Graham Clarke,et al.  A user-independent real-time emotion recognition system for software agents in domestic environments , 2007, Eng. Appl. Artif. Intell..

[36]  Peter C. Fishburn,et al.  Letter to the Editor - Additive Utilities with Incomplete Product Sets: Application to Priorities and Assignments , 1967, Oper. Res..

[37]  R. Plutchik.  The emotions: Facts, theories and a new model. , 1964 .

[38]  Rosalind W. Picard.  Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[39]  Ioanna-Ourania Stathopoulou,et al.  Emotion Recognition from Body Movements and Gestures , 2011 .

[40]  Maria Virvou,et al.  Audio-lingual and Visual-facial Emotion Recognition: Towards a Bi-modal Interaction System , 2010, 2010 22nd IEEE International Conference on Tools with Artificial Intelligence.

[41]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[42]  Ioanna-Ourania Stathopoulou,et al.  Visual Affect Recognition , 2010, Frontiers in Artificial Intelligence and Applications.

[43]  Maria Virvou,et al.  Adapting the human plausible reasoning theory to a graphical user interface , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[44]  Ioanna-Ourania Stathopoulou,et al.  NEU-FACES: A Neural Network-Based Face Image Analysis System , 2007, ICANNGA.

[45]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[46]  Karin Ackermann,et al.  The Nature Of Emotion Fundamental Questions , 2016 .

[47]  Ruili Wang,et al.  Ensemble methods for spoken emotion recognition in call-centres , 2007, Speech Commun..

[48]  Maria Virvou,et al.  Development process of an affective bi-modal Intelligent Tutoring System , 2007, Intell. Decis. Technol..

[49]  J. Stainer,et al.  The Emotions , 1922, Nature.

[50]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[51]  S. Tomkins,et al.  Script theory: differential magnification of affects. , 1978, Nebraska Symposium on Motivation. Nebraska Symposium on Motivation.

[52]  Tsutomu Miyasato,et al.  Multimodal human emotion/expression recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[53]  Felix Naumann.  Data Fusion and Data Quality , 1998 .

[54]  Maria Virvou,et al.  A Knowledge-Based Software Life-Cycle Framework for the Incorporation of Multicriteria Analysis in Intelligent User Interfaces , 2006, IEEE Transactions on Knowledge and Data Engineering.

[55]  Ioanna-Ourania Stathopoulou,et al.  Facial Expression Classification: Specifying Requirements for an Automated System , 2006, KES.

[56]  C. W. Hughes.  Emotion: Theory, Research and Experience , 1982 .

[57]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[58]  Zhiwei Zhu,et al.  Toward a decision-theoretic framework for affect recognition and user assistance , 2006, Int. J. Hum. Comput. Stud..

[59]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[60]  L. de Silva,et al.  Facial emotion recognition using multi-modal information , 1997, Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat..