A component based approach for classifying the seven universal facial expressions of emotion

In a previous publication we showed a component approach performing better than a holistic approach for classifying 6 discrete facial expressions of emotion (happy, sad, surprised, fearful, disgusted and angry) plus the neutral face. In this paper, we present the impact on the two approaches if a seventh facial expression is included. This seventh expression is known as the contemptuous expression (an expression that has rarely been explored in the machine learning literature). Together, the 7 facial expressions are known in the psychology literature as Ekman's Universal Facial Expressions of Emotion. To replicate our previous experiments, we use the same number of image preprocessing techniques (grayscale, Local Binary Patterns, Sobel and Canny edge detection, and manually inserted feature points which we refer to as components) that are organised into four approaches: holistic, holistic action, component and component action (where an action approach is produced by a difference between a neutral and an expressive face). The facial expressions are classified using a standard multiclass Support Vector Machine (SVM) and a pairwise adaptive multiclass SVM (pa-SVM) - which uses pairwise adaptive model parameters. The significance of this study is to provide a thorough understanding of the choices that impact a classifier's performance and the performance dynamics between expression pairs when the contemptuous expression is now considered. In particular, we extend the issues and justify the use of the contemptuous expression by noting its universal presence in human expression and its asymmetric nature, which is not shared with other expressions. Using our face model, which uses feature points around the eyes, brows and mouth, we obtained a best correct classification rate of 98.57% with the contemptuous expression included.

[1]  Frank Y. Shih,et al.  Recognizing facial action units using independent component analysis and support vector machine , 2006, Pattern Recognit..

[2]  W.A. Barrett,et al.  A survey of face recognition algorithms and testing results , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[3]  A. J. Fridlund IS THERE UNIVERSAL RECOGNITION OF EMOTION FROM FACIAL EXPRESSION? A REVIEW OF THE CROSS-CULTURAL STUDIES , 1994 .

[4]  Paul Ekman,et al.  The universality of a contempt expression: A replication , 1988 .

[5]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[6]  Anastasios Tefas,et al.  Salient feature and reliable classifier selection for facial expression classification , 2010, Pattern Recognit..

[7]  Nicu Sebe,et al.  Structural resemblance to emotional expressions predicts evaluation of emotionally neutral faces. , 2009, Emotion.

[8]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, CVPR Workshops.

[9]  Hao Wang,et al.  Affective interaction based on person-independent facial expression space , 2008, Neurocomputing.

[10]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[11]  P. Ekman Emotions revealed , 2004, BMJ.

[12]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[13]  Jun Wang,et al.  Static topographic modeling for facial expression recognition and analysis , 2007, Comput. Vis. Image Underst..

[14]  P. Ekman,et al.  A new pan-cultural facial expression of emotion , 1986 .

[15]  Björn W. Schuller,et al.  Efficient Recognition of Authentic Dynamic Facial Expressions on the Feedtum Database , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[16]  Ying-li Tian,et al.  Evaluation of Face Resolution for Expression Analysis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[17]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[18]  Stephan K. Chalup,et al.  An experimental evaluation of pairwise adaptive support vector machines , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[19]  Stephan K. Chalup,et al.  A face-house paradigm for architectural scene analysis , 2008, CSTST.

[20]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[21]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[22]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[23]  Guodong Guo,et al.  Learning from examples in the small sample case: face expression recognition , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[24]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[25]  Stephan K. Chalup,et al.  A component based approach improves classification of discrete facial expressions over a holistic approach , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[26]  Glenn I. Roisman,et al.  The emotional integration of childhood experience: physiological, facial expressive, and self-reported emotional response during the adult attachment interview. , 2004, Developmental psychology.

[27]  Matti Pietikäinen,et al.  Boosted multi-resolution spatiotemporal descriptors for facial expression recognition , 2009, Pattern Recognit. Lett..

[28]  M. Beszedes,et al.  Comparison of Human and Automatic Facial Emotions and Emotion Intensity Levels Recognition , 2007, 2007 5th International Symposium on Image and Signal Processing and Analysis.

[29]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Shu Liao,et al.  Facial Expression Recognition using Advanced Local Binary Patterns, Tsallis Entropies and Global Appearance Features , 2006, 2006 International Conference on Image Processing.

[31]  Wang Zeng-fu An expression space model for facial expression analysis , 2009 .

[32]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[33]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[34]  David Matsumoto,et al.  More evidence for the universality of a contempt expression , 1992 .

[35]  R. R. Avent,et al.  Machine vision recognition of facial affect using backpropagation neural networks , 1994, Proceedings of 16th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[36]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[38]  Kin-Man Lam,et al.  Facial expression recognition based on shape and texture , 2009, Pattern Recognit..

[39]  Sungsoo Park,et al.  Subtle facial expression recognition using motion magnification , 2009, Pattern Recognit. Lett..

[40]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[41]  Ashok Samal,et al.  Automatic recognition and analysis of human faces and facial expressions: a survey , 1992, Pattern Recognit..

[42]  Fabio Sambataro,et al.  Preferential responses in amygdala and insula during presentation of facial contempt and disgust , 2006, The European journal of neuroscience.

[43]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[44]  S. Kaiser,et al.  Automated coding of facial behavior in human-computer interactions with facs , 1992 .

[45]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[46]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..