Spontaneous Facial Expression Recognition: A Part Based Approach

A part-based approach for spontaneous expression recognition using audio-visual feature and deep convolution neural network (DCNN) is proposed. The ability of convolution neural network to handle variations in translation and scale is exploited for extracting visual features. The sub-regions, namely, eye and mouth parts extracted from the video faces are given as an input to the deep CNN (DCNN) inorder to extract convnet features. The audio features, namely, voice-report, voice intensity, and other prosodic features are used to obtain complementary information useful for classification. The confidence scores of the classifier trained on different facial parts and audio information are combined using different fusion rules for recognizing expressions. The effectiveness of the proposed approach is demonstrated on acted facial expression in wild (AFEW) dataset.

[1]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Proceedings of the 2015 ACM on International Conference on Multimodal Interaction , 2015, ICMI.

[4]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[5]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[6]  Hiroshi Harashima,et al.  Construction and Psychological Evaluation of 3-D Emotion Space , 1995 .

[7]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Hassen Drira,et al.  4-D Facial Expression Recognition by Learning Geometric Deformations , 2014, IEEE Transactions on Cybernetics.

[9]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[10]  Claudia Jiménez-Guarín,et al.  An Affective Inference Model based on Facial Expression Analysis , 2014, IEEE Latin America Transactions.

[11]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[12]  Khashayar Khorasani,et al.  Facial expression recognition using constructive feedforward neural networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Yurong Chen,et al.  Capturing AU-Aware Facial Features and Their Latent Relations for Emotion Recognition in the Wild , 2015, ICMI.

[14]  Larry S. Davis,et al.  Human expression recognition from motion using a radial basis function network architecture , 1996, IEEE Trans. Neural Networks.

[15]  Yoshiaki Tadokoro,et al.  RECOGNITION OF FACIAL EXPRESSIONS USING 2D DCT AND NEURAL NETWORK , 1999 .

[16]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Ralph Gross,et al.  An Image Preprocessing Algorithm for Illumination Invariant Face Recognition , 2003, AVBPA.

[18]  Tamás D. Gedeon,et al.  Collecting Large, Richly Annotated Facial-Expression Databases from Movies , 2012, IEEE MultiMedia.

[19]  Rabab Kreidieh Ward,et al.  A new facial expression recognition technique using 2D DCT and k-means algorithm , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[20]  S. Demleitner.  [Communication without words]. , 1997, Pflege aktuell.

[21]  Shiguang Shan,et al.  Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild , 2014, ICMI.

[22]  Ling Guan,et al.  A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Siu Cheung Hui,et al.  Facial expression recognition from line-based caricatures , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[24]  Qiang Ji,et al.  Simultaneous Facial Feature Tracking and Facial Expression Recognition , 2013, IEEE Transactions on Image Processing.

[25]  Mohammed Bennamoun,et al.  An Automatic Framework for Textured 3D Video-Based Facial Expression Recognition , 2014, IEEE Transactions on Affective Computing.

[26]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[27]  S. Gupta,et al.  Facial expression recognition using facial characteristic points and Gini index , 2012, 2012 Students Conference on Engineering and Systems.

[28]  Razvan Pascanu,et al.  Combining modality specific deep neural networks for emotion recognition in video , 2013, ICMI '13.

[29]  Ninad Thakoor,et al.  Vision and Attention Theory Based Sampling for Continuous Facial Emotion Recognition , 2014, IEEE Transactions on Affective Computing.

[30]  Yajie Tian,et al.  Handbook of face recognition , 2003 .

[31]  Sébastien Marcel,et al.  Lighting Normalization Algorithms for Face Verification , 2005 .