Facial Expression Recognition Algorithm Based on Fusion of Transformed Multilevel Features and Improved Weighted Voting SVM

In allusion to the shortcomings of traditional facial expression recognition (FER) that only uses a single feature and the recognition rate is not high, a FER method based on fusion of transformed multilevel features and improved weighted voting SVM (FTMS) is proposed. The algorithm combines the transformed traditional shallow features and convolutional neural network (CNN) deep semantic features and uses an improved weighted voting method to make a comprehensive decision on the results of the four trained SVM classifiers to obtain the final recognition result. The shallow features include local Gabor features, LBP features, and joint geometric features designed in this study, which are composed of distance and deformation characteristics. The deep feature of CNN is the multilayer feature fusion of CNN proposed in this study. This study also proposes to use a better performance SVM classifier with CNN to replace Softmax since the poor distinction between facial expressions. Experiments on the FERPlus database show that the recognition rate of this method is 17.2% higher than that of the traditional CNN, which proves the effectiveness of the fusion of the multilayer convolutional layer features and SVM. FTMS-based facial expression recognition experiments are carried out on the JAFFE and CK+ datasets. Experimental results show that, compared with the single feature, the proposed algorithm has higher recognition rate and robustness and makes full use of the advantages and characteristics of different features.

[1]  Wei Li,et al.  A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition , 2020, Inf. Fusion.

[2]  Biswajeet Pradhan,et al.  A comparative study of different machine learning methods for landslide susceptibility assessment: A case study of Uttarakhand area (India) , 2016, Environ. Model. Softw..

[3]  Fatima Chouireb,et al.  Classification of textured images based on new information fusion methods , 2019, IET Image Process..

[4]  D. J. Kim,et al.  Facial expression recognition using ASM-based post-processing technique , 2016, Pattern Recognition and Image Analysis.

[5]  Walid Barhoumi,et al.  Sparse coding-based representation of LBP difference for 3D/4D facial expression recognition , 2019, Multimedia Tools and Applications.

[6]  Hong Zhang,et al.  Facial expression recognition via learning deep sparse autoencoders , 2018, Neurocomputing.

[7]  Kaoru Hirota,et al.  Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction , 2018, Inf. Sci..

[8]  Ting Rui,et al.  Convolutional neural network simplification via feature map pruning , 2018, Comput. Electr. Eng..

[9]  Man Lv,et al.  Facial Expression Recognition Based on a Hybrid Model Combining Deep and Shallow Features , 2019, Cognitive Computation.

[10]  Qirong Mao,et al.  Geometry Guided Pose-Invariant Facial Expression Recognition , 2020, IEEE Transactions on Image Processing.

[11]  Hung-Hsu Tsai,et al.  Facial expression recognition using a combination of multiple facial features and support vector machine , 2018, Soft Comput..

[12]  Jing Huang,et al.  Improved Viola-Jones face detection algorithm based on HoloLens , 2019, EURASIP J. Image Video Process..

[13]  Bi Lin,et al.  Automatic recognition and classification of multi-channel microseismic waveform based on DCNN and SVM , 2019, Comput. Geosci..

[14]  Shu Qin,et al.  Facial expression recognition based on Gabor wavelet transform and 2-channel CNN , 2020, Int. J. Wavelets Multiresolution Inf. Process..

[15]  Qirong Mao,et al.  A Unified Deep Model for Joint Facial Expression Recognition, Face Synthesis, and Face Alignment , 2020, IEEE Transactions on Image Processing.

[16]  Yong Wang,et al.  Research on image classification model based on deep convolution neural network , 2019, EURASIP Journal on Image and Video Processing.

[17]  Shan Li,et al.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition , 2019, IEEE Transactions on Image Processing.

[18]  Chen Chen,et al.  Gabor Convolutional Networks , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  Yong Xu,et al.  Image denoising using complex-valued deep CNN , 2021, Pattern Recognit..

[20]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[21]  Jun Zhou,et al.  A method of facial expression recognition based on Gabor and NMF , 2015, Pattern Recognition and Image Analysis.

[22]  Xiaofei Zhou,et al.  Co-saliency detection via integration of multi-layer convolutional features and inter-image propagation , 2020, Neurocomputing.

[23]  Xiaohui Yuan,et al.  Conditional convolution neural network enhanced random forest for facial expression recognition , 2018, Pattern Recognit..

[24]  Burhan Ergen,et al.  Facial emotion recognition on a dataset using convolutional neural network , 2017, 2017 International Artificial Intelligence and Data Processing Symposium (IDAP).

[25]  Zahid Akhtar,et al.  3D palmprint recognition using unsupervised convolutional deep learning network and SVM classifier , 2019, IET Image Process..

[26]  Sanju Mishra,et al.  Facial expression recognition using geometric features and modified hidden Markov model , 2019, Int. J. Grid Util. Comput..

[27]  José Miguel Buenaposada,et al.  Face Alignment using a 3D Deeply-initialized Ensemble of Regression Trees , 2019, Comput. Vis. Image Underst..

[28]  Luping Ji,et al.  Training-Based Gradient LBP Feature Models for Multiresolution Texture Classification. , 2018, IEEE transactions on cybernetics.

[29]  Huadong Li,et al.  Deep reinforcement learning for robust emotional classification in facial expression recognition , 2020, Knowl. Based Syst..

[30]  Fanzhi Kong.  Facial expression recognition method based on deep convolutional neural network combined with improved LBP features , 2019, Personal and Ubiquitous Computing.

[31]  Amjad Rehman,et al.  A Sustainable Deep Learning Framework for Object Recognition Using Multi-Layers Deep Features Fusion and Selection , 2020, Sustainability.

[32]  Jing Tian,et al.  Facial expression classification using salient pattern driven integrated geometric and textual features , 2018, Multimedia Tools and Applications.

[33]  Meng Yang,et al.  Sparse deep feature learning for facial expression recognition , 2019, Pattern Recognit..

[34]  M. Shamim Hossain,et al.  Multilevel Weighted Feature Fusion Using Convolutional Neural Networks for EEG Motor Imagery Classification , 2019, IEEE Access.

[35]  Min Xu,et al.  Manifold feature integration for micro-expression recognition , 2020, Multimedia Systems.

[36]  Patrick Shen-Pei Wang,et al.  Performance Comparisons of Facial Expression Recognition in Jaffe Database , 2008, Int. J. Pattern Recognit. Artif. Intell..

[37]  Nadjiba Terki,et al.  Hierarchical convolutional features for visual tracking via two combined color spaces with SVM classifier , 2018, Signal, Image and Video Processing.

[38]  Armin Lawi,et al.  Facial Expression Recognition using Multiclass Ensemble Least-Square Support Vector Machine , 2018 .

[39]  Zhenyu He,et al.  Joint sparse principal component analysis , 2017, Pattern Recognit..

[40]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[41]  Yuan-Hai Shao,et al.  Joint sample and feature selection via sparse primal and dual LSSVM , 2019, Knowl. Based Syst..

[42]  Shiguang Shan,et al.  Funnel-Structured Cascade for Multi-View Face Detection with Alignment-Awareness , 2016, Neurocomputing.

[43]  Yi Liu,et al.  Facial expression recognition based on Gabor features of salient patches and ACI-LBP , 2018, J. Intell. Fuzzy Syst..