Human emotion and cognition recognition from body language of the head using soft computing techniques

To make a computer interface more usable, enjoyable, and effective, it should be able to recognize emotions of its human counterpart. This paper explores new ways to infer the user’s emotions and cognitions from the combination of facial expression (happy, angry, or sad), eye gaze (direct or averted), and head movement (direction and frequency). All of the extracted information is taken as input data and soft computing techniques are applied to infer emotional and cognitional states. The fuzzy rules were defined based on the opinion of an expert in psychology, a pilot group and annotators. Although the creation of the fuzzy rules are specific to a given culture, the idea of integrating the different modalities of the body language of the head is generic enough to be used by any particular target user group from any culture. Experimental results show that this method can be used to successfully recognize 10 different emotions and cognitions.

[1]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[2]  Mehmet Engin,et al.  ECG beat classification using neuro-fuzzy network , 2004, Pattern Recognit. Lett..

[3]  Trevor Hastie,et al.  Additive Logistic Regression : a Statistical , 1998 .

[4]  Xiaojun Shen,et al.  Facial expression recognition by applying multi-step integral projection and SVMs , 2009, 2009 IEEE Instrumentation and Measurement Technology Conference.

[5]  R. Kleck,et al.  Effects of direct and averted gaze on the perception of facially communicated emotion. , 2005, Emotion.

[6]  Yuxiao Hu,et al.  Audio-visual emotion recognition in adult attachment interview , 2006, ICMI '06.

[7]  Ashish Kapoor,et al.  Multimodal affect recognition in learning environments , 2005, ACM Multimedia.

[8]  Russell Beale,et al.  Affect and Emotion in Human-Computer Interaction, From Theory to Applications , 2008, Affect and Emotion in Human-Computer Interaction.

[9]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[10]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[12]  Tzvi Ganel,et al.  Revisiting the relationship between the processing of gaze direction and the processing of facial expression. , 2011, Journal of experimental psychology. Human perception and performance.

[13]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[14]  Zhiwei Zhu,et al.  Robust Real-Time Face Pose and Facial Expression Recovery , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  K. Scherer,et al.  Multimodal expression of emotion: affect programs or componential appraisal patterns? , 2007, Emotion.

[16]  Lotfi A. Zadeh,et al.  Outline of a New Approach to the Analysis of Complex Systems and Decision Processes , 1973, IEEE Trans. Syst. Man Cybern..

[17]  Hatice Gunes,et al.  Automatic, Dimensional and Continuous Emotion Recognition , 2010, Int. J. Synth. Emot..

[18]  Michael J. Pazzani,et al.  User Modeling for Adaptive News Access , 2000, User Modeling and User-Adapted Interaction.

[19]  Arthur C. Graesser,et al.  Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features , 2010, User Modeling and User-Adapted Interaction.

[20]  Salem Saleh Al-amri.  Linear and Non-linear Contrast Enhancement Image , 2010 .

[21]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[22]  S. Baron-Cohen,et al.  Mind Reading: The Interactive Guide to Emotions , 2003 .

[23]  Massimo Piccardi,et al.  Fusing Face and Body Display for Bi-modal Emotion Recognition: Single Frame Analysis and Multi-frame Post Integration , 2005, ACII.

[24]  Kasia Muldner,et al.  Emotion Sensors Go To School , 2009, AIED.

[25]  G. Feng,et al.  A Survey on Analysis and Design of Model-Based Fuzzy Control Systems , 2006, IEEE Transactions on Fuzzy Systems.

[26]  David Matsumoto,et al.  The Handbook of Culture and Psychology , 2001 .

[27]  Qiang Ji,et al.  A probabilistic framework for modeling and real-time monitoring human fatigue , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[28]  Mehran Jahed,et al.  Real-time intelligent pattern recognition algorithm for surface EMG signals , 2007, Biomedical engineering online.

[29]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[30]  Bernd Kleinjohann,et al.  Real-Time Facial Expression Recognition Using a Fuzzy Emotion Model , 2007, 2007 IEEE International Fuzzy Systems Conference.

[31]  Changsheng Xu,et al.  A Probabilistic Framework for Temporal User Modeling on Microblogs , 2015, CIKM.

[32]  Amit Konar,et al.  Emotion Recognition From Facial Expressions and Its Control Using Fuzzy Logic , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[33]  Maja Pantic,et al.  Fully automatic facial feature point detection using Gabor feature based boosted classifiers , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[34]  E. Brigham,et al.  The fast Fourier transform , 2016, IEEE Spectrum.

[35]  Nicu Sebe,et al.  Multimodal Human Computer Interaction , 2005 .

[36]  Skyler T. Hawk,et al.  Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES). , 2011, Emotion.

[37]  Kostas Karpouzis,et al.  Robust Feature Detection for Facial Expression Recognition , 2007, EURASIP J. Image Video Process..

[38]  Alexander E. Gegov,et al.  Complexity Management in Fuzzy Systems - A Rule Base Compression Approach , 2007, Studies in Fuzziness and Soft Computing.

[39]  C. W. Tao,et al.  A New Neuro-Fuzzy Classifier with Application to On-Line Face Detection and Recognition , 2000, J. VLSI Signal Process..

[40]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[41]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[42]  Christos D. Katsis,et al.  Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[43]  Loïc Kessous,et al.  Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech , 2008, Affect and Emotion in Human-Computer Interaction.

[44]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[45]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[46]  David Matsumoto,et al.  Cultural Influences on the Perception of Emotion , 1989 .

[47]  N. Sadati,et al.  Neuro-Fuzzy Surface EMG Pattern Recognition For Multifunctional Hand Prosthesis Control , 2007, 2007 IEEE International Symposium on Industrial Electronics.

[48]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[49]  Hatice Gunes,et al.  Affect recognition from face and body: early fusion vs. late fusion , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[50]  Reginald B. Adams,et al.  Influence of emotional expression on the processing of gaze direction , 2009 .

[51]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Shrikanth S. Narayanan,et al.  Emotion recognition using a data-driven fuzzy inference system , 2003, INTERSPEECH.

[53]  Hao Shi,et al.  A Novel Neuro Fuzzy Approach to Human Emotion Determination , 2010, 2010 International Conference on Digital Image Computing: Techniques and Applications.

[54]  E. H. Mamdani,et al.  An Experiment in Linguistic Synthesis with a Fuzzy Logic Controller , 1999, Int. J. Man Mach. Stud..

[55]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[57]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[58]  Abigail A Marsh,et al.  Nonverbal “Accents” , 2003, Psychological science.

[59]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[60]  Nicu Sebe,et al.  Multimodal approaches for emotion recognition: a survey , 2005, IS&T/SPIE Electronic Imaging.

[61]  Paolo Remagnino,et al.  From Monomodal to Multimodal: Affect Recognition Using Visual Modalities , 2009 .

[62]  Rosalind W. Picard.  Affective Computing , 1997 .

[63]  K. Scherer,et al.  Emotion recognition from expressions in face, voice, and body: the Multimodal Emotion Recognition Test (MERT). , 2009, Emotion.

[64]  Kostas Karpouzis,et al.  Emotion Analysis in Man-Machine Interaction Systems , 2004, MLMI.

[65]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[66]  D avid M atsumoto,et al.  American ± Japanese Cultural Differences in Judgements of Expression Intensity and Subjective Experience , 2001 .

[67]  Kostas Karpouzis,et al.  Estimation of behavioral user state based on eye gaze and head pose—application in an e-learning environment , 2009, Multimedia Tools and Applications.

[68]  Oleg Starostenko,et al.  Facial Feature Model for Emotion Recognition Using Fuzzy Reasoning , 2010, MCPR.

[69]  Shubhangi D. Giripunje,et al.  ANFIS Based Emotions Recognision in Speech , 2007, KES.

[70]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.