Facial expression recognition via region-based convolutional fusion network

Abstract One of the key challenge issues of deep-learning-based facial expression recognition (FER) is learning effective and robust features from variant samples. In this paper, Region-based Convolutional Fusion Network (RCFN) is proposed to solve this issue via three aspects. Firstly, a muscle movement model is built to segment out crucial regions of frontal face, providing well-unified patches with benefits of removing unrepresentative regions and greatly reducing interference caused by facial organs with varied sizes and positions among individuals. Secondly, a fast and practical network is constructed to extract robust triple-level features from low level to semantic level in each crucial region and fuse them for FER. Thirdly, constrained punitive loss is introduced to leverage the network training for boosting up FER performance. The experiment results show that RCFN is effective in commonly used datasets like KDEF, CK+, and Oulu-CASIA, and can achieve comparable performance with other state-of-the-art FER methods.

[1]  Ping Liu,et al.  Facial Expression Recognition via a Boosted Deep Belief Network , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Mohammed Bennamoun,et al.  A spatio-temporal RBM-based model for facial expression recognition , 2016, Pattern Recognit..

[3]  Xiong Chen,et al.  Facial expression recognition from image sequences using twofold random forest classifier , 2015, Neurocomputing.

[4]  Guihua Wen,et al.  Cognitive facial expression recognition with constrained dimensionality reduction , 2017, Neurocomputing.

[5]  Jiwen Lu,et al.  Simultaneous Local Binary Feature Learning and Encoding for Homogeneous and Heterogeneous Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Shiguang Shan,et al.  Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Sungyoung Lee,et al.  Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields , 2015, IEEE Transactions on Image Processing.

[9]  Xiaoou Tang,et al.  Learning Social Relation Traits from Face Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[10]  Yong Du,et al.  Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks , 2017, IEEE Transactions on Image Processing.

[11]  Haibin Yan,et al.  Collaborative discriminative multi-metric learning for facial expression recognition in video , 2018, Pattern Recognit..

[12]  Radhouane Guermazi,et al.  Facial-expression recognition based on a low-dimensional temporal feature space , 2017, Multimedia Tools and Applications.

[13]  ByoungChul Ko,et al.  A Brief Review of Facial Emotion Recognition Based on Visual Information , 2018, Sensors.

[14]  Anima Majumder,et al.  Automatic Facial Expression Recognition System Using Deep Network-Based Data Fusion , 2018, IEEE Transactions on Cybernetics.

[15]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[16]  Gaurav Sharma,et al.  LOMo: Latent Ordinal Model for Facial Analysis in Videos , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Barnaby Nelson,et al.  Emotion recognition in individuals at clinical high-risk for schizophrenia. , 2012, Schizophrenia bulletin.

[18]  Mohammad Reza Mohammadi,et al.  PCA-based dictionary building for accurate facial expression recognition via sparse representation , 2014, J. Vis. Commun. Image Represent..

[19]  R. Gur,et al.  Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders , 2011, Journal of Neuroscience Methods.

[20]  Matti Pietikäinen,et al.  Facial expression recognition from near-infrared videos , 2011, Image Vis. Comput..

[21]  Zheru Chi,et al.  Facial Expression Recognition in Video with Multiple Feature Fusion , 2018, IEEE Transactions on Affective Computing.

[22]  Hong Zhang,et al.  Facial expression recognition via learning deep sparse autoencoders , 2018, Neurocomputing.

[23]  Jiwen Lu,et al.  Context-Aware Local Binary Feature Learning for Face Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Hyung Il Choi,et al.  An effective facial expression recognition approach for intelligent game systems , 2016, Int. J. Comput. Vis. Robotics.

[25]  Fei Long,et al.  Learning Spatiotemporal and Geometric Features with ISA for Video-Based Facial Expression Recognition , 2017, ICONIP.

[26]  Jiwen Lu,et al.  Learning Rotation-Invariant Local Binary Descriptor , 2017, IEEE Transactions on Image Processing.

[27]  Haitao Zhao,et al.  A visual attention based ROI detection method for facial expression recognition , 2018, Neurocomputing.

[28]  Kin-Man Lam,et al.  Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study , 2018, J. Vis. Commun. Image Represent..

[29]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[30]  Dejey Dharma,et al.  A novel Monogenic Directional Pattern (MDP) and pseudo-Voigt kernel for facilitating the identification of facial emotions , 2017, J. Vis. Commun. Image Represent..

[31]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[32]  Karim Faez,et al.  Facial expression recognition using dual dictionary learning , 2017, J. Vis. Commun. Image Represent..

[33]  Jiwen Lu,et al.  Learning Compact Binary Face Descriptor for Face Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Jiwen Lu,et al.  Discriminative Deep Metric Learning for Face and Kinship Verification , 2017, IEEE Transactions on Image Processing.

[35]  Xia Mao,et al.  Exponential elastic preserving projections for facial expression recognition , 2018, Neurocomputing.

[36]  Kamal Nasrollahi,et al.  Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification , 2017, IEEE Transactions on Cybernetics.