The impact of pre-processing algorithms in facial expression recognition

This paper proposes several pre-processing algorithms to improve facial expression recognition based on Convolutional Neural Networks (CNNs) models. The proposed CNN model was trained on the Extended Cohn-Kanade dataset (CK+) after applying the pre-processing stages and achieved competitive results (93.90% recognition accuracy) despite its simple and light architecture. Using this CNN model, a study on the impact of each pre-processing algorithm when extracting facial features is presented. In the end, it is understood that pre-processing algorithms help CNNs to extract the most relevant features for each facial expression more effectively, reducing the overfitting and increasing the recognition accuracy. Attention maps before and after the pre-processing step are shown in order to visualize its impact when the proposed CNN model makes a prediction.

[1]  D. Lundqvist,et al.  Facial expressions of emotion (KDEF): Identification under different display-duration conditions , 2008, Behavior research methods.

[2]  Priyanka Garg,et al.  A Comparative Study on Histogram Equalization and Cumulative Histogram Equalization , 2017 .

[3]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[4]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[5]  Shariq Hussain,et al.  An Effective Framework for Driver Fatigue Recognition Based on Intelligent Facial Expressions Analysis , 2018, IEEE Access.

[6]  John J. B. Allen,et al.  Frontal EEG asymmetry as a moderator and mediator of emotion , 2004, Biological Psychology.

[7]  Lei He,et al.  Facial Expression Recognition Using Fusion Features Based on Center-Symmetric Local Octonary Pattern , 2019, IEEE Access.

[8]  Xiaoou Tang,et al.  From Facial Expression Recognition to Interpersonal Relation Prediction , 2016, International Journal of Computer Vision.

[9]  António J. R. Neves,et al.  Self-calibration of Colormetric Parameters in Vision Systems for Autonomous Soccer Robots , 2013, RoboCup.

[10]  Nalini Ambady,et al.  Accurate identification of fear facial expressions predicts prosocial behavior. , 2007, Emotion.

[11]  Séverine Dubuisson,et al.  Confidence-Weighted Local Expression Predictions for Occlusion Handling in Expression Recognition and Action Unit Detection , 2016, International Journal of Computer Vision.

[12]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[13]  Jaime S. Cardoso,et al.  Physiological Inspired Deep Neural Networks for Emotion Recognition , 2018, IEEE Access.

[14]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[15]  Thomas S. Huang,et al.  Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition? , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[16]  Abhinav Dhall,et al.  EmotiW 2019: Automatic Emotion, Engagement and Cohesion Prediction Tasks , 2019, ICMI.

[17]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[19]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[20]  Gwen Littlewort,et al.  Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[21]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Dimitris Kanellopoulos,et al.  Data Preprocessing for Supervised Leaning , 2007 .

[23]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[24]  Douglas M. Hawkins,et al.  The Problem of Overfitting , 2004, J. Chem. Inf. Model..

[25]  António J. R. Neves,et al.  Facial Expression Recognition Using Computer Vision: A Systematic Review , 2019, Applied Sciences.

[26]  Alfred M. Bruckstein,et al.  On Gabor's contribution to image enhancement , 1994, Pattern Recognit..

[27]  Heng Wang,et al.  Depression recognition based on dynamic facial and vocal expression features using partial least square regression , 2013, AVEC@ACM Multimedia.

[28]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Tamás D. Gedeon,et al.  Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015 , 2015, ICMI.

[30]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).