Cross-Domain Facial Expression Recognition Based on Transductive Deep Transfer Learning

In this paper, we proposed a novel end-to-end transductive deep transfer learning network (TDTLN) to deal with the challenging cross-domain expression recognition problem, in which both the source and target databases are utilized to jointly learn optimal nonlinear discriminative features so as to improve the label prediction performance of the target data samples. As part of the network parameters, the labels of the target samples are also optimized when optimizing the parameters of TDTLN, such that the cross-entropy loss of source domain data and the regression loss of target domain data can be simultaneously calculated. Finally, to evaluate the recognition performance of the proposed TDTLN method, we conduct extensive cross-database experiments on four commonly used multi-view facial expression databases, namely the BU-3DEF, Multi-PIE, SFEW, and RAF database. The experimental results show that the proposed TDTLN method outperforms state-of-the-art methods.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Thomas S. Huang,et al.  A novel approach to expression recognition from non-frontal face images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[3]  Xiaoyan Zhou,et al.  Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning , 2018, IEEE Transactions on Affective Computing.

[4]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[5]  Guoying Zhao,et al.  Learning a Target Sample Re-Generator for Cross-Database Micro-Expression Recognition , 2017, ACM Multimedia.

[6]  Wenming Zheng,et al.  MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition , 2019, IEEE Access.

[7]  Yuan Shi,et al.  Geodesic flow kernel for unsupervised domain adaptation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  W. Zheng,et al.  Facial expression recognition using kernel canonical correlation analysis (KCCA) , 2006, IEEE Transactions on Neural Networks.

[9]  Xinming Zhang,et al.  A novel framework for semantic segmentation with generative adversarial network , 2019, J. Vis. Commun. Image Represent..

[10]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Guoying Zhao,et al.  Learning From Hierarchical Spatiotemporal Descriptors for Micro-Expression Recognition , 2018, IEEE Transactions on Multimedia.

[12]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[14]  Ling Shao,et al.  Weakly-Supervised Cross-Domain Dictionary Learning for Visual Recognition , 2014, International Journal of Computer Vision.

[15]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[16]  Zhen Cui,et al.  Cross-Database Facial Expression Recognition via Unsupervised Domain Adaptive Dictionary Learning , 2016, ICONIP.

[17]  Tong Zhang,et al.  A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition , 2016, IEEE Transactions on Multimedia.

[18]  Tong Zhang,et al.  Spatial–Temporal Recurrent Neural Network for Emotion Recognition , 2017, IEEE Transactions on Cybernetics.

[19]  Fernando De la Torre,et al.  Selective Transfer Machine for Personalized Facial Expression Analysis , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Tong Zhang,et al.  Unsupervised facial expression recognition using domain adaptation based dictionary learning approach , 2018, Neurocomputing.

[21]  Maja Pantic,et al.  Meta-Analysis of the First Facial Expression Recognition Challenge , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  Richard Bowden,et al.  Local binary patterns for multi-view facial expression recognition , 2011 .

[23]  Thomas S. Huang,et al.  Emotion Recognition from Non-Frontal Facial Images , 2015 .

[24]  Guoying Zhao,et al.  Cross-Database Micro-Expression Recognition: A Benchmark , 2018, IEEE Transactions on Knowledge and Data Engineering.

[25]  Yun Fu,et al.  Semi-supervised Deep Domain Adaptation via Coupled Neural Networks , 2018, IEEE Transactions on Image Processing.

[26]  Wenming Zheng,et al.  Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[27]  Changsheng Li,et al.  Learning Transferable Self-attentive Representations for Action Recognition in Untrimmed Videos with Weak Supervision , 2019, AAAI.

[28]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Xiaoyan Zhou,et al.  Cross-pose color facial expression recognition using transductive transfer linear discriminat analysis , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[31]  Philip S. Yu,et al.  Domain Invariant Transfer Kernel Learning , 2015, IEEE Transactions on Knowledge and Data Engineering.

[32]  Peng Li,et al.  Active semi-supervised learning based on self-expressive correlation with generative adversarial networks , 2019, Neurocomputing.

[33]  Guoying Zhao,et al.  Toward Bridging Microexpressions From Different Domains , 2020, IEEE Transactions on Cybernetics.

[34]  Thomas S. Huang,et al.  Non-frontal view facial expression recognition based on ergodic hidden Markov model supervectors , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[35]  Oscar Déniz-Suárez,et al.  Face recognition using Histograms of Oriented Gradients , 2011, Pattern Recognit. Lett..

[36]  Yoichi Sato,et al.  Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates , 2007, International Journal of Computer Vision.

[37]  Ming Shao,et al.  Graph Adaptive Knowledge Transfer for Unsupervised Domain Adaptation , 2018, ECCV.

[38]  Yang Yang,et al.  Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network , 2019, Neurocomputing.

[39]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[40]  Tinne Tuytelaars,et al.  Unsupervised Visual Domain Adaptation Using Subspace Alignment , 2013, 2013 IEEE International Conference on Computer Vision.

[41]  Pengfei Wei,et al.  Deep Nonlinear Feature Coding for Unsupervised Domain Adaptation , 2016, IJCAI.

[42]  Wenming Zheng,et al.  Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression , 2014, IEEE Transactions on Affective Computing.

[43]  Ivor W. Tsang,et al.  Domain Transfer Multiple Kernel Learning , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Rodrigo Ferreira Berriel,et al.  Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network , 2017, 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).