Dual integrated convolutional neural network for real-time facial expression recognition in the wild

Automatic recognition of facial expressions in the wild is a challenging problem and has drawn a lot of attention from the computer vision and pattern recognition community. Since their emergence, the deep learning techniques have proved their efficacy in facial expression recognition (FER) tasks. However, these techniques are parameter intensive, and thus, could not be deployed on resource-constrained embedded platforms for real-world applications. To mitigate these limitations of the deep learning inspired FER systems, in this paper, we present an efficient dual integrated convolution neural network (DICNN) model for the recognition of facial expressions in the wild in real-time, running on an embedded platform. The designed DICNN model with just 1.08M parameters and 5.40 MB memory storage size achieves optimal performance by maintaining a proper balance between recognition accuracy and computational efficiency. We evaluated the DICNN model on four FER benchmark datasets (FER2013, FERPlus, RAF-DB, and CKPlus) using different performance evaluation metrics, namely the recognition accuracy, precision, recall, and F1-score. Finally, to provide a portable solution with high throughput inference, we optimized the designed DICNN model using TensorRT SDK and deployed it on an Nvidia Xavier embedded platform. Comparative analysis results with the other state-of-the-art methods revealed the effectiveness of the designed FER system, which achieved competitive accuracy with multi-fold improvement in the execution speed.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Shiguang Shan,et al.  Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism , 2019, IEEE Transactions on Image Processing.

[3]  Tzuu-Hseng S. Li,et al.  CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot , 2019, IEEE Access.

[4]  Xia Li,et al.  Deep convolution network based emotion analysis towards mental health care , 2020, Neurocomputing.

[5]  Martin Kampel,et al.  Facial Expression Recognition using Convolutional Neural Networks: State of the Art , 2016, ArXiv.

[6]  Sarangapani Jagannathan,et al.  A comprehensive survey on model compression and acceleration , 2020, Artificial Intelligence Review.

[7]  Zhenqi Li,et al.  SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG , 2019, Front. Neurorobot..

[8]  Minyi Guo,et al.  eXnet: An Efficient Approach for Emotion Recognition in the Wild , 2020, Sensors.

[9]  Priyanka Sharma,et al.  Review of automated emotion-based quantification of facial expression in Parkinson’s patients , 2020, The Visual Computer.

[10]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[11]  Radu Tudor Ionescu,et al.  Local Learning With Deep and Handcrafted Features for Facial Expression Recognition , 2018, IEEE Access.

[12]  Namita Mittal,et al.  Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy , 2019, The Visual Computer.

[13]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[14]  Xiaoou Tang,et al.  Learning Social Relation Traits from Face Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Pourya Shamsolmoali,et al.  Extended deep neural network for facial emotion recognition , 2019, Pattern Recognit. Lett..

[16]  Yongxin Zhu,et al.  Recognizing Facial Expressions Using a Shallow Convolutional Neural Network , 2019, IEEE Access.

[17]  Andrew Zisserman,et al.  Emotion Recognition in Speech using Cross-Modal Transfer in the Wild , 2018, ACM Multimedia.

[18]  Xiaoqian Liu,et al.  Improved curriculum learning using SSM for facial expression recognition , 2019, The Visual Computer.

[19]  Christina Huang,et al.  Combining convolutional neural networks for emotion recognition , 2017, 2017 IEEE MIT Undergraduate Research Technology Conference (URTC).

[20]  Bin Huang,et al.  Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture , 2020, Pattern Recognit. Lett..

[21]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Ying Chen,et al.  Video-Based Facial Expression Recognition using Deep Temporal–Spatial Networks , 2020, IETE Technical Review.

[23]  Fengping An,et al.  Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM , 2019, The Visual Computer.

[24]  Haifeng Hu,et al.  Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition , 2019, Pattern Recognit..

[25]  Maie Bachmann,et al.  Audiovisual emotion recognition in wild , 2018, Machine Vision and Applications.

[26]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[27]  Luca Fanucci,et al.  An FPGA-Based Hardware Accelerator for CNNs Using On-Chip Memories Only: Design and Benchmarking with Intel Movidius Neural Compute Stick , 2019, Int. J. Reconfigurable Comput..

[28]  Emad Barsoum,et al.  Training deep networks for facial expression recognition with crowd-sourced label distribution , 2016, ICMI.

[29]  ByoungChul Ko,et al.  Driver’s Facial Expression Recognition in Real-Time for Safe Driving , 2018, Sensors.

[30]  Xinran Yu,et al.  Heavy path based super-sequence frequent pattern mining on web log dataset , 2015, Artif. Intell. Res..

[31]  Jian Zhang,et al.  Learning deep facial expression features from image and optical flow sequences using 3D CNN , 2018, The Visual Computer.

[32]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[33]  Jianfei Yang,et al.  Suppressing Uncertainties for Large-Scale Facial Expression Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Soo-Hyung Kim,et al.  Facial Emotion Recognition Using an Ensemble of Multi-Level Convolutional Neural Networks , 2019, Int. J. Pattern Recognit. Artif. Intell..

[35]  Ying Zeng,et al.  EEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations , 2019, IEEE Transactions on Biomedical Engineering.

[36]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Zheng Lian,et al.  Region Based Robust Facial Expression Analysis , 2018, 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia).

[38]  Shan Li,et al.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition , 2019, IEEE Transactions on Image Processing.

[39]  Jianfei Yang,et al.  Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition , 2019, IEEE Transactions on Image Processing.

[40]  Ming Li,et al.  Facial Expression Recognition with Identity and Emotion Joint Learning , 2018, IEEE Transactions on Affective Computing.

[41]  Guihua Wen,et al.  Sample awareness-based personalized facial expression recognition , 2019, Applied Intelligence.

[42]  Giancarlo Fortino,et al.  Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network , 2017, IEEE Access.

[43]  Yueli Cui,et al.  Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning , 2019, IEEE Access.

[44]  Vivienne Sze,et al.  Efficient Processing of Deep Neural Networks: A Tutorial and Survey , 2017, Proceedings of the IEEE.

[45]  Dong Keun Kim,et al.  The Design of CNN Architectures for Optimal Six Basic Emotion Classification Using Multiple Physiological Signals , 2020, Sensors.

[46]  Min Yu,et al.  Expression Recognition Method Based on a Lightweight Convolutional Neural Network , 2020, IEEE Access.

[47]  Jörgen Ahlberg,et al.  Fast facial expression recognition using local binary features and shallow neural networks , 2018, The Visual Computer.

[48]  Rongrong Ni,et al.  Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images , 2018, IEEE Access.

[49]  Jinzhao Wu,et al.  Deep Temporal-Spatial Aggregation for Video-Based Facial Expression Recognition , 2019, Symmetry.

[50]  Jianfeng Zhao,et al.  Speech emotion recognition using deep 1D & 2D CNN LSTM networks , 2019, Biomed. Signal Process. Control..

[51]  Soo-Young Lee,et al.  Hierarchical committee of deep convolutional neural networks for robust facial expression recognition , 2016, Journal on Multimodal User Interfaces.

[52]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[53]  Jie Shao,et al.  Three convolutional neural network models for facial expression recognition in the wild , 2019, Neurocomputing.

[54]  Hasan Demirel,et al.  3D CNN-Based Speech Emotion Recognition Using K-Means Clustering and Spectrograms , 2019, Entropy.

[55]  Yi Jin,et al.  Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy , 2020, The Visual Computer.

[56]  Elad Eban,et al.  MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[57]  Ram Mohana Reddy Guddeti,et al.  Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks , 2019, Education and Information Technologies.