Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images

In this paper, we propose a spontaneous facial expression recognition method by using feature-level fusion of visible and thermal infrared facial images. Firstly, the appearance features of visible images and statistic parameters of thermal infrared difference images are extracted. Then, analysis of variance is adopted to select the optimal feature subsets from both visible and thermal ones. These selected features are combined as the input of a K-Nearest Neighbors classifier. We experimentally evaluate the effectiveness of the proposed method on USTC-NVIE database. The experimental results show that fusion of visible and thermal infrared features can improve the accuracy rate of negative expressions and reduce the discrepancy. Thus, it can improve the expression recognition performance.

[1]  Saurabh Singh,et al.  Face recognition by fusing thermal infrared and visible imagery , 2006, Image Vis. Comput..

[2]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[3]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Rana El Kaliouby,et al.  When Human Coders (and Machines) Disagree on the Meaning of Facial Affect in Spontaneous Videos , 2009, IVA.

[5]  Dirk Van Compernolle,et al.  Feature subset selection for improved native accent identification , 2010, Speech Commun..

[6]  Yasunari Yoshitomi,et al.  Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face , 2000, Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499).

[7]  Michael Beetz,et al.  Model Based Analysis of Face Images for Facial Feature Extraction , 2009, CAIP.

[8]  Nicu Sebe,et al.  Authentic Facial Expression Analysis , 2004, FGR.

[9]  Kwang-Eun Ko,et al.  Development of the facial feature extraction and emotion recognition method based on ASM and Bayesian network , 2009, 2009 IEEE International Conference on Fuzzy Systems.

[10]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Masood Mehmood Khan,et al.  Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature , 2009, TAP.

[12]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[14]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Arthur C. Graesser,et al.  Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features , 2010, User Modeling and User-Adapted Interaction.