Systematic review of 3D facial expression recognition methods

Abstract The three-dimensional representation of the human face has emerged as a viable and effective way to characterize the facial surface for expression classification purposes. The rapid progress in the area continually demands its up-to-date characterization to guide and support research decisions, specially for newcomer researchers. This systematic literature review focus on investigating three major aspects of 3D facial expression recognition methods: face representation, preprocessing and classification experiments. The investigation of 49 specialized studies revealed the preferential types of data and regions of interest for face representation in recent years, as well as a trend towards keypoint-independent methods. In addition, it brings to light current weaknesses regarding the report of preprocessing techniques and identifies challenges concerning the current possibility of fair comparison among multiple methods. The presented findings outline essential research decisions whose the regardful report is of great value to this research community.

[1]  Ahmed Maalej,et al.  3D facial expression recognition using nonrigid CPD registration method , 2017, 2017 International Conference on Information and Digital Technologies (IDT).

[2]  Arman Savran,et al.  Computer Vision and Image Understanding , 2022, SSRN Electronic Journal.

[3]  Xing Zhang,et al.  Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[4]  Qiuqi Ruan,et al.  3D Facial expression recognition using orthogonal tensor marginal fisher analysis on geometric maps , 2017, 2017 International Conference on Wavelet Analysis and Pattern Recognition (ICWAPR).

[5]  Hasan Demirel,et al.  Feature selection for improved 3D facial expression recognition , 2014, Pattern Recognit. Lett..

[6]  Hedi Tabia,et al.  Dynamic 3D Facial Expression Recognition Using Robust Shape Features , 2013, SCIA.

[7]  Mehryar Emambakhsh,et al.  Nasal Patches and Curves for Expression-Robust 3D Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Wei Zeng,et al.  An automatic 3D expression recognition framework based on sparse representation of conformal images , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[9]  Arsalane Zarghili,et al.  A $P recognizer for automatic facial emotion recognition using Kinect sensor , 2017, 2017 Intelligent Systems and Computer Vision (ISCV).

[10]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[11]  Ibrahim Venkat,et al.  Identifying universal facial emotion markers for automatic 3D facial expression recognition , 2014, 2014 International Conference on Computer and Information Sciences (ICCOINS).

[12]  Mohammad H. Mahoor,et al.  Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  David Declercq,et al.  3D facial expression recognition using kernel methods on Riemannian manifold , 2017, Eng. Appl. Artif. Intell..

[14]  Mohammed Bennamoun,et al.  Clustering of video-patches on Grassmannian manifold for facial expression recognition from 3D videos , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[15]  Yunhong Wang,et al.  Automatic 4D Facial Expression Recognition Using Dynamic Geometrical Image Network , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[16]  Shikha Tripathi,et al.  Pose invariant method for emotion recognition from 3D images , 2015, 2015 Annual IEEE India Conference (INDICON).

[17]  Alberto Del Bimbo,et al.  Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans , 2013, The Visual Computer.

[18]  Nicu Sebe,et al.  Facial expression recognition under a wide range of head poses , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[19]  Jian Sun,et al.  Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network , 2017, IEEE Transactions on Multimedia.

[20]  Ling Li,et al.  Fully automatic 3D facial expression recognition using local depth features , 2014, IEEE Winter Conference on Applications of Computer Vision.

[21]  Stefanos Zafeiriou,et al.  A dynamic approach to the recognition of 3D facial expressions and their temporal models , 2011, Face and Gesture 2011.

[22]  Liming Chen,et al.  Muscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition , 2015, IEEE Transactions on Multimedia.

[23]  Bogdan J. Matuszewski,et al.  Is 2D Unlabeled Data Adequate for Recognizing Facial Expressions? , 2016, IEEE Intelligent Systems.

[24]  Giancarlo Fortino,et al.  A facial expression recognition system using robust face features from depth videos and deep learning , 2017, Comput. Electr. Eng..

[25]  Yiheng Cai,et al.  3D facial expression recognition using distance features and LBP features based on automatically detected keypoints , 2016, 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[26]  Ibrahim Venkat,et al.  Analysis and evaluation of SURF descriptors for automatic 3D facial expression recognition using different classifiers , 2014, 2014 4th World Congress on Information and Communication Technologies (WICT 2014).

[27]  Xi Zhao,et al.  An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition , 2015, Comput. Vis. Image Underst..

[28]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[29]  Hasan Demirel,et al.  Feature Selection for Enhanced 3D Facial Expression Recognition Based on Varying Feature Point Distances , 2013, ISCIS.

[30]  A. Lynn Abbott,et al.  A multi-modal feature fusion framework for kinect-based facial expression recognition using Dual Kernel Discriminant Analysis (DKDA) , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[31]  Hasan Demirel,et al.  Fuzzy SVM for 3D facial expression classification using sequential forward feature selection , 2017, 2017 9th International Conference on Computational Intelligence and Communication Networks (CICN).

[32]  Hassen Drira,et al.  4-D Facial Expression Recognition by Learning Geometric Deformations , 2014, IEEE Transactions on Cybernetics.

[33]  A. Zarghili,et al.  Automatic recognition of facial expressions using microsoft kinect with artificial neural network , 2016, 2016 International Conference on Engineering & MIS (ICEMIS).

[34]  Karim Faez,et al.  2D facial expression recognition via 3D reconstruction and feature fusion , 2016, J. Vis. Commun. Image Represent..

[35]  Yunhong Wang,et al.  Texture and Geometry Scattering Representation-Based Facial Expression Recognition in 2D+3D Videos , 2018, ACM Trans. Multim. Comput. Commun. Appl..

[36]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[37]  Q. M. Jonathan Wu,et al.  A survey of local feature methods for 3D face recognition , 2017, Pattern Recognit..

[38]  Fernando Fernández Martínez,et al.  Towards a robust affect recognition: Automatic facial expression recognition in 3D faces , 2015, Expert Syst. Appl..

[39]  Li Zhang,et al.  Adaptive 3D facial action intensity estimation and emotion recognition , 2015, Expert Syst. Appl..

[40]  Antonios Danelakis,et al.  A survey on facial expression recognition in 3D video sequences , 2014, Multimedia Tools and Applications.

[41]  Ling Li,et al.  Automatic 4D Facial Expression Recognition Using DCT Features , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[42]  A. Lynn Abbott,et al.  Adaptive feature selection and data pruning for 3D facial expression recognition using the Kinect , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[43]  P D Bamidis,et al.  Affective Medicine , 2010, Methods of Information in Medicine.

[44]  Federico Sukno,et al.  Local Shape Spectrum Analysis for 3D Facial Expression Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[45]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Lijun Yin,et al.  Static and dynamic 3D facial expression recognition: A comprehensive survey , 2012, Image Vis. Comput..

[47]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[48]  Hassen Drira,et al.  Magnifying Subtle Facial Motions for Effective 4D Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[49]  Liming Chen,et al.  Automatic 3D facial expression recognition using geometric scattering representation , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[50]  Ting Zhang,et al.  Facial Expression Recognition Based on Deep Learning: A Survey , 2017 .

[51]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[52]  Ioannis A. Kakadiaris,et al.  3D Face Discriminant Analysis Using Gauss-Markov Posterior Marginals , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Guilherme Horta Travassos,et al.  Scientific research ontology to support systematic review in software engineering , 2007, Adv. Eng. Informatics.

[54]  Asim Jan,et al.  Automatic 3D facial expression recognition using geometric and textured feature fusion , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[55]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[56]  Qiuqi Ruan,et al.  3D facial expression recognition algorithm using local threshold binary pattern and histogram of oriented gradient , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[57]  Adnan Khashman,et al.  Intelligent recognition of emotional expressions in 3D face images , 2013, 2013 21st Signal Processing and Communications Applications Conference (SIU).

[58]  Liming Chen,et al.  Fast and Light Manifold CNN based 3D Facial Expression Recognition across Pose Variations , 2018, ACM Multimedia.

[59]  Liming Chen,et al.  Author manuscript, published in "Workshop 3D Face Biometrics, IEEE Automatic Facial and Gesture Recognition, Shanghai: China (2013)" Fully Automatic 3D Facial Expression Recognition using Differential Mean Curvature Maps and Histograms of Oriented Gradien , 2013 .

[60]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[61]  Björn W. Schuller,et al.  Detection of security related affect and behaviour in passenger transport , 2008, INTERSPEECH.

[62]  Qiuqi Ruan,et al.  Fast nonnegative tensor factorization based on graph-preserving for 3D facial expression recognition , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[63]  Dan Han,et al.  Facial expression recognition with LBP and SLPP combined method , 2014, 2014 12th International Conference on Signal Processing (ICSP).

[64]  A. Lynn Abbott,et al.  VT-KFER: A Kinect-based RGBD+time dataset for spontaneous and non-spontaneous facial expression recognition , 2015, 2015 International Conference on Biometrics (ICB).

[65]  Khan M. Iftekharuddin,et al.  Frenet Frame-Based Generalized Space Curve Representation for Pose-Invariant Classification and Recognition of 3-D Face , 2016, IEEE Transactions on Human-Machine Systems.

[66]  Xiaoou Tang,et al.  Automatic facial expression recognition on a single 3D face by exploring shape deformation , 2009, ACM Multimedia.

[67]  Hasan Demirel,et al.  Entropy-based feature selection for improved 3D facial expression recognition , 2013, Signal, Image and Video Processing.

[68]  Qiuqi Ruan,et al.  Multiple strategies to enhance automatic 3D facial expression recognition , 2015, Neurocomputing.

[69]  Hasan Demirel,et al.  Person Independent Facial Expression Recognition Using 3D Facial Feature Positions , 2012, ISCIS.