A Review on Facial Expression Recognition: Feature Extraction and Classification

ABSTRACT Facial expression recognition (FER) is currently a very active research topic in the fields of computer vision, pattern recognition, artificial intelligence, and has drawn extensive attentions owing to its potential applications to natural human–computer interaction (HCI), human emotion analysis, interactive video, image indexing and retrieval, etc. This paper is a survey of FER addressing the most two important aspects of designing an FER system. The first one is facial feature extraction for static images and dynamic image sequences. The second one is facial expression classification. Conclusions and future work are finally discussed in the last section of this survey.

[1]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[2]  Y. V. Venkatesh,et al.  Facial expression recognition using radial encoding of local Gabor features and classifier synthesis , 2012, Pattern Recognit..

[3]  Surendra Ranganath,et al.  Cloud basis function neural network: A modified RBF network architecture for holistic facial expression recognition , 2008, Pattern Recognit..

[4]  Hasan Demirel,et al.  Feature selection for improved 3D facial expression recognition , 2014, Pattern Recognit. Lett..

[5]  Mark A. Clements,et al.  Multimodal Affect Classification at Various Temporal Lengths , 2015, IEEE Transactions on Affective Computing.

[6]  Nicu Sebe,et al.  Authentic facial expression analysis , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[7]  Zhengyou Zhang,et al.  Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[8]  Lijun Yin,et al.  Static and dynamic 3D facial expression recognition: A comprehensive survey , 2012, Image Vis. Comput..

[9]  Ana Belén Moreno,et al.  Differential optical flow applied to automatic facial expression recognition , 2011, Neurocomputing.

[10]  Shahriar Negahdaripour,et al.  Revised Definition of Optical Flow: Integration of Radiometric and Geometric Cues for Dynamic Scene Analysis , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Yan-tao Tian,et al.  Facial Expression Recognition Method Based on Gabor Wavelet Features and Fractional Power Polynomial Kernel PCA , 2010, ISNN.

[12]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[13]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[14]  Xuelong Li,et al.  GA-SIFT: A new scale invariant feature transform for multispectral image using geometric algebra , 2014, Inf. Sci..

[15]  Yongzhao Zhan,et al.  A neural-AdaBoost based facial expression recognition system , 2014, Expert Syst. Appl..

[16]  Takeo Kanade,et al.  Subtly different facial expression recognition and expression intensity estimation , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[17]  Ali N. Akansu,et al.  Facial expression recognition with regional hidden Markov models , 2014 .

[18]  Emmanuel Dellandréa,et al.  A unified probabilistic framework for automatic 3D facial expression analysis based on a Bayesian belief inference and statistical feature models , 2013, Image Vis. Comput..

[19]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[20]  Shiqing Zhang,et al.  Facial expression recognition using local binary patterns and discriminant kernel locally linear embedding , 2012, EURASIP Journal on Advances in Signal Processing.

[21]  Zhiyong Feng,et al.  Facial expression recognition via deep learning , 2014, 2014 International Conference on Smart Computing.

[22]  Claudio A. Perez,et al.  Face recognition under pose variation with local Gabor features enhanced by Active Shape and Statistical Models , 2015, Pattern Recognit..

[23]  Mohammad Reza Mohammadi,et al.  PCA-based dictionary building for accurate facial expression recognition via sparse representation , 2014, J. Vis. Commun. Image Represent..

[24]  Di Huang,et al.  Local Binary Patterns and Its Application to Facial Image Analysis: A Survey , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[25]  Oksam Chae,et al.  Robust Facial Expression Recognition Based on Local Directional Pattern , 2010 .

[26]  Dong Yu,et al.  Deep Learning and Its Applications to Signal and Information Processing , 2011 .

[27]  Daijin Kim,et al.  Natural facial expression recognition using differential-AAM and manifold learning , 2009, Pattern Recognit..

[28]  Nong Sang,et al.  Accurate and robust facial expressions recognition by fusing multiple sparse representation based classifiers , 2015, Neurocomputing.

[29]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[30]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Shiqing Zhang,et al.  Facial Expression Recognition Using Sparse Representation , 2012 .

[32]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Ling Guan,et al.  A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[34]  Hasan Demirel,et al.  Facial expression recognition based on discriminative scale invariant feature transform , 2010 .

[35]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Stefanos Zafeiriou,et al.  HOG active appearance models , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[37]  Fernando De la Torre,et al.  Facial Expression Analysis , 2011, Visual Analysis of Humans.

[38]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[39]  Uipil Chong,et al.  Facial Expression Recognition Using Local Transitional Pattern on Gabor Filtered Facial Images , 2013 .

[40]  Jean-Philippe Thiran,et al.  Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data , 2015, Pattern Recognit. Lett..

[41]  Qiuqi Ruan,et al.  Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns , 2015, Signal Process..

[42]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[43]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[44]  Shiqing Zhang,et al.  Robust Facial Expression Recognition via Compressive Sensing , 2012, Sensors.

[45]  Deepak Ghimire,et al.  Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines , 2013, Sensors.

[46]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[47]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[49]  Xiaohu Wang,et al.  Multimodal Emotion Recognition Integrating Affective Speech with Facial Expression , 2014 .

[50]  Shikun Zhou,et al.  Facial Expression Analysis using Active Shape Model , 2015 .

[51]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[52]  Björn Stenger,et al.  Using Bounded Diameter Minimum Spanning Trees to Build Dense Active Appearance Models , 2014, International Journal of Computer Vision.

[53]  Qiang Ji,et al.  A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  S. Demleitner.  [Communication without words]. , 1997, Pflege aktuell.

[55]  Ying Chen,et al.  Regression-based Active Appearance Model initialization for facial feature tracking with missing frames , 2014, Pattern Recognit. Lett..

[56]  Shiqing Zhang,et al.  Facial expression recognition based on local binary patterns and local fisher discriminant analysis , 2012 .

[57]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[58]  Alberto Del Bimbo,et al.  A Set of Selected SIFT Features for 3D Facial Expression Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[59]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[60]  Khashayar Khorasani,et al.  Facial expression recognition using constructive feedforward neural networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[61]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[62]  Sungyoung Lee,et al.  Depth Camera-Based Facial Expression Recognition System Using Multilayer Scheme , 2014 .

[63]  Dong Yu,et al.  Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP] , 2011, IEEE Signal Processing Magazine.

[64]  Arman Savran,et al.  Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities , 2015, IEEE Transactions on Cybernetics.

[65]  George A. Papakostas,et al.  Lattice Computing Extension of the FAM Neural Classifier for Human Facial Expression Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Catalin-Daniel Caleanu.  Face expression recognition: A brief overview of the last decade , 2013, 2013 IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI).

[67]  Min Chen,et al.  Facial expression recognition in dynamic sequences: An integrated approach , 2014, Pattern Recognit..