Multi-task Micro-expression Recognition Combining Deep and Handcrafted Features

Micro-expression recognition is a challenging problem due to its short duration and low intensity. Most previous work on micro-expression mainly used the handcrafted features. Recently, deep learning methods were also employed for some difficult face recognition tasks. This paper presents a new framework to recognize micro-expression by combining handcrafted features and deep features. The employed handcrafted feature is called Local Gabor Binary Pattern from Three Orthogonal Panels (LGBP-TOP) feature. LGBP-TOP combines spatial and temporal analysis to encode the local facial movements. The employed deep feature is based on the Convolutional Neural Network (CNN) model trained on the micro-expression dataset. And then, the sparse multi-task learning framework with adaptive penalty term is employed to remove the irrelevant information from the combined LGBP-TOP and CNN features. The experimental evaluation is performed on two widely used micro-expression databases. The results demonstrate that the proposed approach achieves a competitive performance compared with other popular micro-expression recognition methods.

[1]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[2]  Stéphane Canu,et al.  $\ell_{p}-\ell_{q}$ Penalty for Sparse Linear and Sparse Multiple Kernel Multitask Learning , 2011, IEEE Transactions on Neural Networks.

[3]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[4]  Marian Stewart Bartlett,et al.  Multilayer Architectures for Facial Action Unit Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[6]  Rui Yan,et al.  Micro-expression recognition system , 2016 .

[7]  Yantao Tian,et al.  Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[8]  John See,et al.  LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition , 2014, ACCV.

[9]  Shuicheng Yan,et al.  Visual classification with multi-task joint sparse representation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Huicheng Zheng,et al.  A Delaunay-Based Temporal Coding Model for Micro-expression Recognition , 2014, ACCV Workshops.

[11]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[12]  Xi Chen,et al.  Accelerated Gradient Method for Multi-task Sparse Learning Problem , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[13]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[14]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[15]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[16]  Michel F. Valstar,et al.  Local Gabor Binary Patterns from Three Orthogonal Planes for Automatic Facial Expression Recognition , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[17]  Shang Gao,et al.  A new sparse representation-based classification algorithm using iterative class elimination , 2013, Neural Computing and Applications.

[18]  R. Vidal,et al.  Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[20]  Wei-Shi Zheng,et al.  Multi-task mid-level feature learning for micro-expression recognition , 2017, Pattern Recognit..

[21]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[22]  Zhiyong Feng,et al.  Facial expression recognition via deep learning , 2014, 2014 International Conference on Smart Computing.

[23]  Bingpeng Ma,et al.  Robust Head Pose Estimation Using LGBP , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[24]  Ming Yu,et al.  Micro-expression recognition based on CBP-TOP feature with ELM , 2015 .

[25]  Guoying Zhao,et al.  Selective deep features for micro-expression recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[26]  Matti Pietikäinen,et al.  Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[27]  Yong Man Ro,et al.  Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations , 2016, ACM Multimedia.

[28]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[30]  Peng Zhang,et al.  Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation , 2016, Neural Computing and Applications.

[31]  Hao Zheng.  Micro-Expression Recognition based on 2D Gabor Filter and Sparse Representation , 2017 .

[32]  Guoying Zhao,et al.  Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space , 2014, 2014 22nd International Conference on Pattern Recognition.

[33]  Miao Tian,et al.  Deep Belief Network Based on Double Weber Local Descriptor in Micro-expression Recognition , 2017, MUE/FutureTech.

[34]  Xin Geng,et al.  A Relaxed K-SVD Algorithm for Spontaneous Micro-Expression Recognition , 2016, PRICAI.

[35]  KokSheik Wong,et al.  Optical strain based recognition of subtle emotions , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).