Fusion of visible and thermal images for facial expression recognition

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.

[1]  Ioannis T. Pavlidis,et al.  Thermal image analysis for anxiety detection , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[2]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[4]  Paul Choon Keat Lee,et al.  Optical Emission Spectroscopy to study FeCo thin film deposition using plasma focus , 2005 .

[5]  Chris H. Q. Ding,et al.  Analysis of gene expression profiles: class discovery and leaf ordering , 2002, RECOMB '02.

[6]  Lv Shan-wei.  The Research Advance of Facial Expression Recognition in Human Computer Interaction , 2009 .

[7]  Hatice Gunes,et al.  Dimensional Emotion Prediction from Spontaneous Head Gestures for Interaction with Sensitive Artificial Listeners , 2010, IVA.

[8]  Qiang Ji,et al.  A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Kamaruzzaman Sopian,et al.  Proceedings of the 11th WSEAS international conference on Applied computer science , 2009 .

[10]  Ioannis T. Pavlidis,et al.  StressCam: non-contact measurement of users' emotional states through thermal imaging , 2005, CHI Extended Abstracts.

[11]  Zhaoyu Wang,et al.  Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images , 2011, 2011 IEEE International Workshop on Machine Learning for Signal Processing.

[12]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[13]  Masood Mehmood Khan,et al.  Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature , 2009, TAP.

[14]  Shangfei Wang,et al.  Facial Expression Recognition from Infrared Thermal Videos , 2012, IAS.

[15]  Yang Wang,et al.  Robust facial feature tracking under varying face pose and facial expression , 2007, Pattern Recognit..

[16]  Gwen Littlewort,et al.  The motion in emotion — A CERT based approach to the FERA emotion challenge , 2011, Face and Gesture 2011.

[17]  David Sander,et al.  Thermal Analysis of Facial Muscles Contractions , 2011, IEEE Transactions on Affective Computing.

[18]  Rocco A. Servedio,et al.  Discriminative learning can succeed where generative learning fails , 2006, Inf. Process. Lett..

[19]  Stan Z. Li,et al.  Advances in Biometrics, International Conference, ICB 2007, Seoul, Korea, August 27-29, 2007, Proceedings , 2007, ICB.

[20]  Yasunari Yoshitomi,et al.  Facial expression recognition for speaker using thermal image processing and speech recognition system , 2010 .

[21]  Fei Chen,et al.  A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference , 2010, IEEE Transactions on Multimedia.

[22]  Shangfei Wang,et al.  A spontaneous facial expression recognition method using head motion and AAM features , 2010, 2010 Second World Congress on Nature and Biologically Inspired Computing (NaBIC).

[23]  Javier Ruiz-del-Solar,et al.  A comparative study of thermal face recognition methods in unconstrained environments , 2012, Pattern Recognit..

[24]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[25]  Shan He,et al.  Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images , 2012, IAS.

[26]  Vinay Bettadapura,et al.  Face Expression Recognition and Analysis: The State of the Art , 2012, ArXiv.

[27]  Zhang Yu-jin.  New Research Advances in Facial Expression Recognition , 2006 .

[28]  Chih-Jen Lin,et al.  Probability Estimates for Multi-class Classification by Pairwise Coupling , 2003, J. Mach. Learn. Res..

[29]  Yasunari Yoshitomi,et al.  Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face , 2000, Proceedings 9th IEEE International Workshop on Robot and Human Interactive Communication. IEEE RO-MAN 2000 (Cat. No.00TH8499).

[30]  Pradeep Buddharaju,et al.  A Comparative Analysis of Thermal and Visual Modalities for Automated Facial Expression Recognition , 2012, ISVC.

[31]  Leonardo Trujillo,et al.  Visual learning of texture descriptors for facial expression recognition in thermal imagery , 2007, Comput. Vis. Image Underst..

[32]  Jing Xiao,et al.  Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[33]  Wang Zhiliang.  Survey of Facial Expression Recognition Based on Computer Vision , 2006 .

[34]  Pradeep Buddharaju,et al.  Face Recognition Beyond the Visible Spectrum , 2008 .

[35]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.