Adaptive Feature Mapping for Customizing Deep Learning Based Facial Expression Recognition Model

Automated facial expression recognition can greatly improve the human–machine interface. The machine can provide better and more personalized services when it knows the human’s emotion. This kind of improvement is an important progress in this artificial intelligence era. Many deep learning approaches have been applied in recent years due to their outstanding recognition accuracy after training with large amounts of data. The performance is limited, however, by the specific environmental conditions and variations in different persons involved. Hence, this paper addresses the issue of how to customize the generic model without label information from the testing samples. Weighted Center Regression Adaptive Feature Mapping (W-CR-AFM) is mainly proposed to transform the feature distribution of testing samples into that of trained samples. By means of minimizing the error between each feature of testing sample and the center of the most relevant category, W-CR-AFM can bring the features of testing samples around the decision boundary to the centers of expression categories; therefore, their predicted labels can be corrected. When the model which is tuned by W-CR-AFM is tested on extended Cohn-Kanade (CK+), Radboud Faces database, and Amsterdam dynamic facial expression set, our approach can improve the recognition accuracy by about 3.01%, 0.49%, and 5.33%, respectively. Compared to the competing deep learning architectures with the same training data, our approach shows the better performance.

[1]  Qirong Mao,et al.  Hierarchical Bayesian Theme Models for Multipose Facial Expression Recognition , 2017, IEEE Transactions on Multimedia.

[2]  Zhaoqiang Xia,et al.  Towards Facial Expression Recognition in the Wild: A New Database and Deep Recognition System , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Hamid Sadeghi,et al.  Facial expression recognition using geometric normalization and appearance representation , 2013, 2013 8th Iranian Conference on Machine Vision and Image Processing (MVIP).

[4]  Skyler T. Hawk,et al.  Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES). , 2011, Emotion.

[5]  Jun Yu,et al.  Deep Neural Networks with Relativity Learning for facial expression recognition , 2016, 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[6]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Keun-Chang Kwak,et al.  Facial Expression Recognition Using 3D Convolutional Neural Network , 2014 .

[8]  Dong-Chen He,et al.  Texture Unit, Texture Spectrum, And Texture Analysis , 1990 .

[9]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[10]  Baoqing Li,et al.  Facial Expression Recognition From Image Sequence Based on LBP and Taylor Expansion , 2017, IEEE Access.

[11]  Tong Zhang,et al.  A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition , 2016, IEEE Transactions on Multimedia.

[12]  Junmo Kim,et al.  Development of deep learning-based facial expression recognition system , 2015, 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV).

[13]  Qijun Zhao,et al.  Discriminative Feature Adaptation for cross-domain facial expression recognition , 2016, 2016 International Conference on Biometrics (ICB).

[14]  Davis E. King.  Max-Margin Object Detection , 2015, ArXiv.

[15]  Yu-Chiang Frank Wang,et al.  Unsupervised Domain Adaptation with Imbalanced Cross-Domain Data , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  Wei Li,et al.  A deep-learning approach to facial expression recognition with candid images , 2015, 2015 14th IAPR International Conference on Machine Vision Applications (MVA).

[17]  Byungyong Ryu,et al.  Local Directional Ternary Pattern for Facial Expression Recognition. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[18]  Tal Hassner,et al.  Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns , 2015, ICMI.

[19]  Thai Hoang Le,et al.  Deep generic features and SVM for facial expression recognition , 2016, 2016 3rd National Foundation for Science and Technology Development Conference on Information and Computer Science (NICS).

[20]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[21]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[22]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[23]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Yongzhao Zhan,et al.  Pose-robust feature learning for facial expression recognition , 2015, Frontiers of Computer Science.

[25]  Ping Liu,et al.  Identity-Aware Convolutional Neural Network for Facial Expression Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[26]  Stefan Wermter,et al.  Face expression recognition with a 2-channel Convolutional Neural Network , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[27]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[28]  Oliver G. B. Garrod,et al.  Facial expressions of emotion are not culturally universal , 2012, Proceedings of the National Academy of Sciences.

[29]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[30]  Mohammad H. Mahoor,et al.  Facial Expression Recognition from World Wild Web , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[31]  Jiwen Lu,et al.  Cost-Sensitive Local Binary Feature Learning for Facial Age Estimation , 2015, IEEE Transactions on Image Processing.

[32]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[33]  Junmo Kim,et al.  Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Edilson de Aguiar,et al.  A Facial Expression Recognition System Using Convolutional Networks , 2015, 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images.

[35]  Fernando De la Torre,et al.  Selective Transfer Machine for Personalized Facial Expression Analysis , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.