Deep Emotion Transfer Network for Cross-database Facial Expression Recognition

Due to the large domain discrepancy between the training and testing data and the inaccessibility of annotating sufficient training samples, cross-database facial expression recognition which has more application value remains to be challenging in the literature. Previous researches on this problem are based on shallow features with limited discrimination ability. In this paper, we propose to address this problem with a Deep Emo-transfer Network (DETN). Specifically, maximum mean discrepancy was embedded in the deep architecture to reduce dataset bias. Furthermore, a very common but widely ignored bottleneck in facial expression, imbalanced class distribution, has been taken into account. A learnable class-wise weighting parameter was introduced to our network by exploring class prior distribution on unlabeled data so that the training and testing domains can share similar class distribution. Extensive empirical evidences involving both lab-controlled vs. real-world and small-scale vs. large-scale facial expression databases show that our DETN can yield competitive performances across various facial expression transfer tasks.

[1]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[2]  Mohammad H. Mahoor,et al.  Facial expression recognition using lp\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${l}_{p}$$\end{document}-norm MKL , 2015, Machine Vision and Applications.

[3]  Mohammad H. Mahoor,et al.  Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[4]  Danyang Li,et al.  Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition , 2017, Cognitive Computation.

[5]  B. Radig,et al.  Cross-database evaluation for facial expression recognition , 2014, Pattern Recognition and Image Analysis.

[6]  Stefan Winkler,et al.  Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning , 2015, ICMI.

[7]  Tae-Sun Choi,et al.  Boosted NNE collections for multicultural facial expression recognition , 2016, Pattern Recognit..

[8]  Zhen Cui,et al.  Cross-Database Facial Expression Recognition via Unsupervised Domain Adaptive Dictionary Learning , 2016, ICONIP.

[9]  Hélio Pedrini,et al.  Effects of cultural characteristics on building an emotion classifier through facial expression analysis , 2015, J. Electronic Imaging.

[10]  Marcelo H. Ang,et al.  Cross-dataset facial expression recognition , 2011, 2011 IEEE International Conference on Robotics and Automation.

[11]  Y. V. Venkatesh,et al.  Facial expression recognition using radial encoding of local Gabor features and classifier synthesis , 2012, Pattern Recognit..

[12]  Junmo Kim,et al.  Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[13]  Ninad Thakoor,et al.  One shot emotion scores for facial emotion recognition , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[14]  Mohammad H. Mahoor,et al.  Facial Expression Recognition Using Enhanced Deep 3 D Convolutional Neural Networks , .

[15]  Jianhou Gan,et al.  Feature Extraction based on Local Directional Pattern with SVM Decision-level Fusion for Facial Expression Recognition , 2013 .

[16]  Gwen Littlewort,et al.  Toward Practical Smile Detection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[18]  Tamás D. Gedeon,et al.  Emotion recognition using PHOG and LPQ features , 2011, Face and Gesture 2011.

[19]  Martin D. Levine,et al.  Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers , 2014, IEEE Transactions on Affective Computing.

[20]  Guoying Zhao,et al.  Learning a Target Sample Re-Generator for Cross-Database Micro-Expression Recognition , 2017, ACM Multimedia.

[21]  Junping Du,et al.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[23]  Rodrigo Ferreira Berriel,et al.  Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network , 2017, 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).

[24]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[25]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[26]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, CVPR Workshops.

[27]  Yoshua Bengio,et al.  Semi-supervised Learning by Entropy Minimization , 2004, CAP.

[28]  Guoying Zhao,et al.  Facial Affect “In-the-Wild”: A Survey and a New Database , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Mohamed S. Kamel,et al.  Cross-Domain Facial Expression Recognition Using Supervised Kernel Mean Matching , 2012, 2012 11th International Conference on Machine Learning and Applications.

[30]  Qijun Zhao,et al.  Discriminative Feature Adaptation for cross-domain facial expression recognition , 2016, 2016 International Conference on Biometrics (ICB).

[31]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[32]  Shiguang Shan,et al.  AU-inspired Deep Networks for Facial Expression Feature Learning , 2015, Neurocomputing.

[33]  Tamás D. Gedeon,et al.  Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[34]  M. Pantic,et al.  Induced Disgust , Happiness and Surprise : an Addition to the MMI Facial Expression Database , 2010 .

[35]  Cha Zhang,et al.  Image based Static Facial Expression Recognition with Multiple Deep Network Learning , 2015, ICMI.

[36]  Qilong Wang,et al.  Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Laurens van der Maaten,et al.  Accelerating t-SNE using tree-based algorithms , 2014, J. Mach. Learn. Res..

[38]  Hans-Peter Kriegel,et al.  Integrating structured biological data by Kernel Maximum Mean Discrepancy , 2006, ISMB.

[39]  Yoshua Bengio,et al.  Challenges in representation learning: A report on three machine learning contests , 2013, Neural Networks.

[40]  Sivaraman Balakrishnan,et al.  Optimal kernel choice for large-scale two-sample tests , 2012, NIPS.

[41]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.