Recognition of 3D emotional facial expression based on handcrafted and deep feature combination

Abstract Facial emotion recognition (FER) methods have been proposed mainly using 2D images. These methods suffer from many problems caused by the difficult conditions of unconstrained environments such as light conditions and view variations. In this paper, we aim to recognize the emotional facial expressions independently of their identity using the 3D data and 2D depth images. Since the 3D FER is a very fine-grained recognition task, mapping the 3D images into 2D depth images may lack some geometric characteristics of the expressive face and decay the FER performance. Convolutional Neural Networks (CNN), however, have been successfully applied to the 2D depth images and improved handcrafted-based methods in computer vision and pattern recognition applications. For this reason, we combine in this paper two types of features; handcrafted and deep learning features and prove their complementarity for 3D FER. Favorably, covariance descriptors have proven a very good ability to combine features from different types into a compact representation. Therefore, we propose to use the covariance matrices of features (handcrafted and deep ones), instead of the features independently. Since covariance matrices belong to one of the manifold space types, formed by SPD (Symmetric Positive Definite) matrices, we mainly focus on the generalization of the RBF kernel to the manifold space for 3D FER using a supervised SVM classification. The achieved performance of the proposed method on the Bosphorus and BU-3DFE datasets outperforms similar state-of-the-arts.

[1]  Soo-Hyung Kim,et al.  Facial Emotion Recognition Using an Ensemble of Multi-Level Convolutional Neural Networks , 2019, Int. J. Pattern Recognit. Artif. Intell..

[2]  S. Selva Nidhyananthan,et al.  3D Facial Expression Recognition Using Multi-channel Deep Learning Framework , 2020, Circuits Syst. Signal Process..

[3]  Fernando Fernández Martínez,et al.  Towards a robust affect recognition: Automatic facial expression recognition in 3D faces , 2015, Expert Syst. Appl..

[4]  Junbin Gao,et al.  Kernel Sparse Subspace Clustering on Symmetric Positive Definite Manifolds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Danilo Comminiello,et al.  Compressing deep quaternion neural networks with targeted regularization , 2019, CAAI Trans. Intell. Technol..

[6]  Barbara Caputo,et al.  Recognition with local features: the kernel recipe , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[7]  W. R. Sam Emmanuel,et al.  A Survey on Human Face Expression Recognition Techniques , 2018, J. King Saud Univ. Comput. Inf. Sci..

[8]  Alessandro Lameiras Koerich,et al.  Facial expression recognition using a pairwise feature selection and classification approach , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[9]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[10]  Nikolaos Papanikolopoulos,et al.  Covariance based point cloud descriptors for object detection and recognition , 2016, Comput. Vis. Image Underst..

[11]  Vipul K. Dabhi,et al.  Survey on Face Expression Recognition using CNN , 2019, 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS).

[12]  Shan Liu,et al.  Neural saliency algorithm guide bi-directional visual perception style transfer , 2020, CAAI Trans. Intell. Technol..

[13]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[14]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[15]  Irene Y. H. Gu,et al.  Exploiting riemannian manifolds for daily activity classification in video towards health care , 2016, 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom).

[16]  Hongdong Li,et al.  Kernel Methods on the Riemannian Manifold of Symmetric Positive Definite Matrices , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Liming Chen,et al.  Fast and Light Manifold CNN based 3D Facial Expression Recognition across Pose Variations , 2018, ACM Multimedia.

[18]  Yong Man Ro,et al.  Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition , 2014, IEEE Transactions on Affective Computing.

[19]  Janusz Konrad,et al.  Action Recognition From Video Using Feature Covariance Matrices , 2013, IEEE Transactions on Image Processing.

[20]  Chun Chen,et al.  Feature level analysis for 3D facial expression recognition , 2011, Neurocomputing.

[21]  Luc Van Gool,et al.  Covariance Pooling for Facial Expression Recognition , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Liming Chen,et al.  Unsupervised Domain Adaptation with Regularized Optimal Transport for Multimodal 2D+3D Facial Expression Recognition , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[23]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[24]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[25]  Yiding Wang,et al.  Learning Encoded Facial Curvature Information for 3D Facial Emotion Recognition , 2013, 2013 Seventh International Conference on Image and Graphics.

[26]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[27]  Sang-Heon Lee,et al.  Facial Expression Recognition Using Extended Local Binary Patterns of 3D Curvature , 2013, MUE.

[28]  Xuan-Phung Huynh,et al.  Convolutional Neural Network Models for Facial Expression Recognition Using BU-3DFE Database , 2016 .

[29]  Dimitrios Hatzinakos,et al.  Iterative Closest Normal Point for 3D Face Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Alberto Del Bimbo,et al.  A Set of Selected SIFT Features for 3D Facial Expression Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[31]  Qiuqi Ruan,et al.  FERLrTc: 2D+3D facial expression recognition via low-rank tensor completion , 2019, Signal Process..

[32]  Guoying Zhao,et al.  3D Facial Expression Recognition Based on Multi-View and Prior Knowledge Fusion , 2019, 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP).

[33]  Titouan Parcollet,et al.  A survey of quaternion neural networks , 2019, Artificial Intelligence Review.

[34]  Mehrtash Tafazzoli Harandi,et al.  From Manifold to Manifold: Geometry-Aware Dimensionality Reduction for SPD Matrices , 2014, ECCV.

[35]  Hasan Demirel,et al.  Optimal feature selection for 3D facial expression recognition using coarse-to-fine classification , 2010 .

[36]  W. Hariri,et al.  Efficient Graph-based Kernel using Covariance Descriptors for 3D Facial Expression Classification , 2020 .

[37]  Ioannis A. Kakadiaris,et al.  3D/4D facial expression analysis: An advanced annotated face model approach , 2012, Image Vis. Comput..

[38]  Gueesang Lee,et al.  A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition , 2019, Image Vis. Comput..

[39]  Mark Elshaw,et al.  Stacked deep convolutional auto-encoders for emotion recognition from facial expressions , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[40]  Lijun Yin,et al.  Identity-Adaptive Facial Expression Recognition through Expression Regeneration Using Conditional Generative Adversarial Networks , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[41]  Federico Sukno,et al.  Local Shape Spectrum Analysis for 3D Facial Expression Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).