Efficient facial expression recognition using histogram of oriented gradients in wavelet domain

Facial expression recognition plays a significant role in human behavior detection. In this study, we present an efficient and fast facial expression recognition system. We introduce a new feature called W_HOG where W indicates discrete wavelet transform and HOG indicates histogram of oriented gradients feature. The proposed framework comprises of four stages: (i) Face processing, (ii) Domain transformation, (iii) Feature extraction and (iv) Expression recognition. Face processing is composed of face detection, cropping and normalization steps. In domain transformation, spatial domain features are transformed into the frequency domain by applying discrete wavelet transform (DWT). Feature extraction is performed by retrieving Histogram of Oriented Gradients (HOG) feature in DWT domain which is termed as W_HOG feature. For expression recognition, W_HOG feature is supplied to a well-designed tree based multiclass support vector machine (SVM) classifier with one-versus-all architecture. The proposed system is trained and tested with benchmark CK+, JAFFE and Yale facial expression datasets. Experimental results of the proposed method are effective towards facial expression recognition and outperforms existing methods.

[1]  Shiguang Shan,et al.  Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis , 2014, ACCV.

[2]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[3]  Inchul Song,et al.  Deep learning for real-time robust facial expression recognition on a smartphone , 2014, 2014 IEEE International Conference on Consumer Electronics (ICCE).

[4]  Alexei A. Efros,et al.  Ensemble of exemplar-SVMs for object detection and beyond , 2011, 2011 International Conference on Computer Vision.

[5]  Junmo Kim,et al.  Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Md. Zia Uddin,et al.  An enhanced independent component-based human facial expression recognition from video , 2009, IEEE Transactions on Consumer Electronics.

[7]  Lyle H. Ungar,et al.  Beyond Binary Labels: Political Ideology Prediction of Twitter Users , 2017, ACL.

[8]  M. Taner Eskil,et al.  Facial expression recognition based on anatomy , 2014, Comput. Vis. Image Underst..

[9]  Maja Pantic,et al.  Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and tools , 2013, Image Vis. Comput..

[10]  Swati Nigam,et al.  Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences , 2016, Multimedia Tools and Applications.

[11]  David S. Rosenblum,et al.  From action to activity: Sensor-based activity recognition , 2016, Neurocomputing.

[12]  Raymond Chiong.  Intelligent Systems for Automated Learning and Adaptation: Emerging Trends and Applications , 2010, Intelligent Systems for Automated Learning and Adaptation.

[13]  M. Seetha,et al.  Subspace based Expression Recognition Using Combinational Gabor based Feature Fusion , 2017 .

[14]  Zhengyou Zhang,et al.  Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with A Multilayer Perceptron , 1999, Int. J. Pattern Recognit. Artif. Intell..

[15]  Tardi Tjahjadi,et al.  A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences , 2015, Pattern Recognit..

[16]  Alice Caplier,et al.  Spontaneous Facial Expression Recognition using Sparse Representation , 2017, VISIGRAPP 2017.

[17]  Alexandros Iosifidis,et al.  Approximate kernel extreme learning machine for large scale data classification , 2017, Neurocomputing.

[18]  Junmo Kim,et al.  Development of deep learning-based facial expression recognition system , 2015, 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV).

[19]  Anastasios Tefas,et al.  Subclass discriminant Nonnegative Matrix Factorization for facial image analysis , 2012, Pattern Recognit..

[20]  Zhiyong Feng,et al.  Facial expression recognition via deep learning , 2014, 2014 International Conference on Smart Computing.

[21]  Nobuyuki Otsu,et al.  Facial expression recognition using Fisher weight maps , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[22]  D. Adam.  The illustrated wavelet transform handbook: introductory theory and applications in science, engineering, medicine and finance , 2004 .

[23]  Swati Nigam,et al.  Multiresolution approach for multiple human detection using moments and local binary patterns , 2014, Multimedia Tools and Applications.

[24]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[25]  Sridha Sridharan,et al.  Person-independent facial expression detection using Constrained Local Models , 2011, Face and Gesture 2011.

[26]  Guodong Guo,et al.  Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[29]  Luming Zhang,et al.  Action2Activity: Recognizing Complex Activities from Sensor Data , 2015, IJCAI.

[30]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[31]  Tardi Tjahjadi,et al.  A dynamic framework based on local Zernike moment and motion history image for facial expression recognition , 2017, Pattern Recognit..

[32]  Maja Pantic,et al.  Facial Expression Recognition , 2009, Encyclopedia of Biometrics.

[33]  Yu Zheng,et al.  Urban Water Quality Prediction Based on Multi-Task Multi-View Learning , 2016, IJCAI.

[34]  Ayyaz Hussain,et al.  Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features , 2017, Multimedia Tools and Applications.

[35]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[36]  Ping Liu,et al.  Identity-Aware Convolutional Neural Network for Facial Expression Recognition , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[37]  J. Russell,et al.  The science of facial expression. , 2017 .

[38]  Akriti Jaiswal,et al.  Facial Emotion Detection Using Deep Learning , 2020, 2020 International Conference for Emerging Technology (INCET).

[39]  A. Treves,et al.  A neural network facial expression recognition system using unsupervised local processing , 2001, ISPA 2001. Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis. In conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat..

[40]  Shiguang Shan,et al.  Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[42]  Swati Nigam,et al.  Towards Intelligent Human Behavior Detection for Video Surveillance , 2019, Censorship, Surveillance, and Privacy.

[43]  Swati Nigam,et al.  Multiscale Local Binary Patterns for Facial Expression-Based Human Emotion Recognition , 2015 .

[44]  Wei Li,et al.  A deep-learning approach to facial expression recognition with candid images , 2015, 2015 14th IAPR International Conference on Machine Vision Applications (MVA).

[45]  Ahmed Bouridane,et al.  Emotion recognition from scrambled facial images via many graph embedding , 2017, Pattern Recognit..

[46]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[47]  Ming-Hsuan Yang,et al.  Kernel Eigenfaces vs. Kernel Fisherfaces: Face recognition using kernel methods , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[48]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[49]  Daniel Llatas Spiers.  Facial emotion detection using deep learning , 2016 .

[50]  Tae-Sun Choi,et al.  Boosted NNE collections for multicultural facial expression recognition , 2016, Pattern Recognit..

[51]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[53]  Manasi Patil,et al.  Facial Expression Recognition Based on Image Feature , 2012 .

[54]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[55]  Shamik Tiwari,et al.  Wavelet Based Histogram of Oriented Gradients Feature Descriptors for Classification of Partially Occluded Objects , 2015 .

[56]  Maja Pantic,et al.  Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[57]  Shiqing Zhang,et al.  A Review on Facial Expression Recognition: Feature Extraction and Classification , 2016 .

[58]  Jian Yang,et al.  Generalized K-L transform based combined feature extraction , 2002, Pattern Recognit..

[59]  Ping Liu,et al.  Facial Expression Recognition via a Boosted Deep Belief Network , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Marcus Liwicki,et al.  DeXpression: Deep Convolutional Neural Network for Expression Recognition , 2015, ArXiv.

[61]  Erik Cambria,et al.  A review of affective computing: From unimodal analysis to multimodal fusion , 2017, Inf. Fusion.

[62]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[63]  Maja Pantic,et al.  Coupled Gaussian processes for pose-invariant facial expression recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Luming Zhang,et al.  Fortune Teller: Predicting Your Career Path , 2016, AAAI.

[65]  Li Liu,et al.  Recognizing Complex Activities by a Probabilistic Interval-Based Model , 2016, AAAI.

[66]  Michael J. Lyons,et al.  Evidence and a computational explanation of cultural differences in facial expression recognition. , 2010, Emotion.

[67]  Takeo Kanade,et al.  Facial Expression Recognition , 2011, Handbook of Face Recognition.

[68]  Qiang Ji,et al.  Capturing Complex Spatio-temporal Relations among Facial Muscles for Facial Expression Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[69]  Shiguang Shan,et al.  AU-inspired Deep Networks for Facial Expression Feature Learning , 2015, Neurocomputing.

[70]  Ming-Hsuan Yang,et al.  Kernel Eigenfaces vs. Kernel Fisherfaces: Face recognition using kernel methods , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.