Analysis of hand-crafted and learned feature extraction methods for real-time facial expression recognition

This paper presents an analysis of hand-crafted and learned feature extraction methods for real-time facial expression recognition (FER). Our analysis focuses on methods capable of running on mobile devices, including traditional algorithms such as Gabor transform, HOG, LBP, as well as two compact CNN models, named Mobilenet V1 and V2. Additionally, we test the performance of MOTIF, a highly efficient texture feature extractor algorithm. Furthermore, we analyze the contribution of the mouth and front-eyes regions for recognizing the seven basic facial expressions. Experimental results are evaluated on two publicly available datasets. KDEF database which was captured under controlled conditions and RAF database which represents more naturalistic expressions captured in-the-wild. Under the same experimental conditions, MOTIF presents the fastest performance by sacrificing accuracy, while Mobilenet V2 presents the highest results with considerable speed and model size.

[1]  Shan Li,et al.  Deep Facial Expression Recognition: A Survey , 2018, IEEE Transactions on Affective Computing.

[2]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Chengjun Liu,et al.  Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition , 2002, IEEE Trans. Image Process..

[5]  Masahide Kaneko,et al.  Multicultural Facial Expression Recognition Based on Differences of Western-Caucasian and East-Asian Facial Expressions of Emotions , 2018, IEICE Trans. Inf. Syst..

[6]  Sang Hyun Park,et al.  Facial expression recognition based on local region specific features and support vector machines , 2016, Multimedia Tools and Applications.

[7]  D. Lundqvist,et al.  Karolinska Directed Emotional Faces , 2015 .

[8]  Masahide Kaneko,et al.  Facial Expression Recognition Based on Facial Region Segmentation and Modal Value Approach , 2014, IEICE Trans. Inf. Syst..

[9]  Hyunseung Choo,et al.  Evaluating real-life performance of the state-of-the-art in facial expression recognition using a novel YouTube-based datasets , 2016, Multimedia Tools and Applications.

[10]  Héctor M. Pérez Meana,et al.  Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach , 2016, Knowl. Based Syst..

[11]  Junping Du,et al.  Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[13]  L. Sumalatha,et al.  Content based Image Retrieval Using Multi Motif Co-Occurrence Matrix , 2018 .

[14]  Farshad Tajeripour,et al.  Texture classification using optimal Gabor filters , 2011 .

[15]  Jesus Olivares-Mercado,et al.  Face recognition system based on MOTIF features , 2018 .

[16]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[18]  Takeo Kanade,et al.  Facial Expression Recognition , 2011, Handbook of Face Recognition.

[19]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[20]  Quan Wen,et al.  Deep Learning Models for Facial Expression Recognition , 2018, 2018 Digital Image Computing: Techniques and Applications (DICTA).

[21]  W. R. Sam Emmanuel,et al.  A Survey on Human Face Expression Recognition Techniques , 2018, J. King Saud Univ. Comput. Inf. Sci..

[22]  Manasi S. Patwardhan,et al.  Survey on real-time facial expression recognition techniques , 2016, IET Biom..

[23]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).