Micro-expression recognition: an updated review of current trends, challenges and solutions

Micro-expression (ME) recognition has attracted numerous interests within the computer vision circle in different contexts particularly, localization, magnification, and recognition. Challenges in these areas remain relevant due to the nature of ME’s split-second transition with minute intensity levels. In this paper, a comprehensive state-of-the-art analysis of ME recognition and detection challenges are provided. Contemporary solutions are categorized into low-level, mid-level, and high-level solutions with a review of their characteristics and performances. This paper also provides possible extensions to basic methods, highlight, and predict emerging trends. A thorough analysis of mainstream ME datasets is also provided by elucidating each of their advantages and limitations. This survey gives readers an understanding of ME recognition and an appreciation of future research direction in ME recognition systems.

[1]  Matti Pietikäinen,et al.  Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis , 2014, 2014 22nd International Conference on Pattern Recognition.

[2]  Andrea Cavallaro,et al.  Learning Bases of Activity for Facial Expression Recognition , 2017, IEEE Transactions on Image Processing.

[3]  Moi Hoon Yap,et al.  Micro-Facial Movement Detection Using Individualised Baselines and Histogram-Based Descriptors , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[4]  Xin Zhang,et al.  Learning effective binary descriptors for micro-expression recognition transferred by macro-information , 2017, Pattern Recognit. Lett..

[5]  Chengjun Liu,et al.  Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition , 2002, IEEE Trans. Image Process..

[6]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[7]  Dmitry B. Goldgof,et al.  Automatic expression spotting in videos , 2014, Image Vis. Comput..

[8]  Hubert Konik,et al.  Micro-Expression Spotting Using the Riesz Pyramid , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[9]  Hui Wu,et al.  Scene text detection using adaptive color reduction, adjacent character model and hybrid verification strategy , 2015, The Visual Computer.

[10]  Yaser Sheikh,et al.  Hand Keypoint Detection in Single Images Using Multiview Bootstrapping , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Aurobinda Routray,et al.  Fuzzy Histogram of Optical Flow Orientations for Micro-Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[12]  Huicheng Zheng,et al.  A Delaunay-Based Temporal Coding Model for Micro-expression Recognition , 2014, ACCV Workshops.

[13]  Guoying Zhao,et al.  Micro-Expression Spotting: A Benchmark , 2017, ArXiv.

[14]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[15]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[16]  Qiang Wu,et al.  A survey: facial micro-expression recognition , 2017, Multimedia Tools and Applications.

[17]  Anhar Risnumawan,et al.  A Scene Image is Nonmutually Exclusive—A Fuzzy Qualitative Scene Understanding , 2014, IEEE Transactions on Fuzzy Systems.

[18]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[19]  Robert H. Riffenburgh,et al.  Linear Discriminant Analysis , 1960 .

[20]  Dmitry B. Goldgof,et al.  Towards macro- and micro-expression spotting in video using strain patterns , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[21]  Guoying Zhao,et al.  Spontaneous micro-expression spotting via geometric deformation modeling , 2016, Comput. Vis. Image Underst..

[22]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[23]  Min Xu,et al.  Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets , 2017, 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[24]  Truong Q. Nguyen,et al.  Real-Time Affine Global Motion Estimation Using Phase Correlation and its Application for Digital Image Stabilization , 2011, IEEE Transactions on Image Processing.

[25]  John See,et al.  LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition , 2014, ACCV.

[26]  Wen-Jing Yan,et al.  How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions , 2013 .

[27]  Matti Pietikäinen,et al.  Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[28]  Rama Chellappa,et al.  A deep pyramid Deformable Part Model for face detection , 2015, 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[29]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[30]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[31]  John See,et al.  Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application , 2016, IEEE Transactions on Affective Computing.

[32]  T. Wheatley,et al.  Emotions Are Understood From Biological Motion Across Remote Cultures , 2017, Emotion.

[33]  Matti Pietikäinen,et al.  Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods , 2015, IEEE Transactions on Affective Computing.

[34]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[35]  Honghai Liu,et al.  Fuzzy Qualitative Trigonometry , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[36]  Yi Jin,et al.  Multi-task feature learning-based improved supervised descent method for facial landmark detection , 2018, Signal Image Video Process..

[37]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[38]  Min Peng,et al.  Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition , 2017, Front. Psychol..

[39]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[40]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Qiang Shen,et al.  Fuzzy qualitative simulation , 1993, IEEE Trans. Syst. Man Cybern..

[42]  Ming Yu,et al.  Micro-expression recognition based on CBP-TOP feature with ELM , 2015 .

[43]  John See,et al.  Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions , 2014, ACCV.

[44]  Guoying Zhao,et al.  Quantifying Micro-expressions with Constraint Local Model and Local Binary Pattern , 2014, ECCV Workshops.

[45]  Andrea Cavallaro,et al.  Robust Registration of Dynamic Facial Sequences , 2017, IEEE Transactions on Image Processing.

[46]  Guoying Zhao,et al.  Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space , 2014, 2014 22nd International Conference on Pattern Recognition.

[47]  Shuo Yang,et al.  From Facial Parts Responses to Face Detection: A Deep Learning Approach , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[48]  Kam Meng Goh,et al.  Fuzzy qualitative approach for micro-expression recognition , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[49]  Georgios D. Evangelidis,et al.  Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Iyanu Pelumi Adegun,et al.  Automatic recognition of micro-expressions using local binary patterns on three orthogonal planes and extreme learning machine , 2016, 2016 Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech).

[51]  Shengcai Liao,et al.  A Fast and Accurate Unconstrained Face Detector , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[53]  Jie Zhu,et al.  Effective micro-expression recognition using relaxed K-SVD algorithm , 2017, Int. J. Mach. Learn. Cybern..

[54]  Radhika M. Pai,et al.  Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences , 2016, 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[55]  Gang Hua,et al.  A convolutional neural network cascade for face detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Yuichi Ohta,et al.  Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS) , 2013, IEICE Trans. Inf. Syst..

[57]  Matti Pietikäinen,et al.  Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns , 2016, Neurocomputing.

[58]  Georgios Tzimiropoulos,et al.  Project-Out Cascaded Regression with an application to face alignment , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Wei Pan,et al.  An Adaptable-Multilayer Fractional Fourier Transform Approach for Image Registration , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Moi Hoon Yap,et al.  Objective Micro-Facial Movement Detection Using FACS-Based Regions and Baseline Evaluation , 2016, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[61]  Matti Pietikäinen,et al.  Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[62]  Xiaolan Fu,et al.  CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition , 2018, IEEE Transactions on Affective Computing.

[63]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  S. Gunn.  Support Vector Machines for Classification and Regression , 1998 .

[65]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[66]  Yong Man Ro,et al.  Using color texture sparsity for facial expression recognition , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[67]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[68]  John See,et al.  Monogenic Riesz wavelet representation for micro-expression recognition , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[69]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Peter Bull,et al.  Detecting Deception from Emotional and Unemotional Cues , 2009 .

[71]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[72]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[74]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[75]  Masakazu Matsugu,et al.  Subject independent facial expression recognition with robust face detection using a convolutional neural network , 2003, Neural Networks.

[76]  KokSheik Wong,et al.  Spontaneous Subtle Expression Detection and Recognition based on Facial Strain , 2016, Signal Process. Image Commun..

[77]  Qingshan Liu,et al.  Dual Sparse Constrained Cascade Regression for Robust Face Alignment , 2016, IEEE Transactions on Image Processing.

[78]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[79]  David Dagan Feng,et al.  Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation , 2017, The Visual Computer.

[80]  Peng Zhang,et al.  Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation , 2016, Neural Computing and Applications.

[81]  Marian Stewart Bartlett,et al.  Exploring Bag of Words Architectures in the Facial Expression Domain , 2012, ECCV Workshops.

[82]  Nicholas Costen,et al.  SAMM: A Spontaneous Micro-Facial Movement Dataset , 2018, IEEE Transactions on Affective Computing.

[83]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[84]  Guangcan Liu,et al.  Deeper cascaded peak-piloted network for weak expression recognition , 2018, The Visual Computer.

[85]  Guoying Zhao,et al.  Selective deep features for micro-expression recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[86]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[87]  KokSheik Wong,et al.  Less is More: Micro-expression Recognition from Video using Apex Frame , 2016, Signal Process. Image Commun..

[88]  Yong Man Ro,et al.  Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[89]  Rajendran Parthiban,et al.  Spatiotemporal feature extraction for facial expression recognition , 2016, IET Image Process..

[90]  B. D. Lucas.  Generalized image matching by the method of differences , 1985 .

[91]  Guoying Zhao,et al.  Sliding Window Based Micro-expression Spotting: A Benchmark , 2017, ACIVS.

[92]  Mao Ye,et al.  Action recognition by learning temporal slowness invariant features , 2016, The Visual Computer.

[93]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[94]  P. Ekman,et al.  Facial action coding system , 2019 .

[95]  Matti Pietikäinen,et al.  Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[96]  Weihong Deng,et al.  Facial landmark localization by enhanced convolutional neural network , 2018, Neurocomputing.

[97]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[98]  Wei-Shi Zheng,et al.  Multi-task mid-level feature learning for micro-expression recognition , 2017, Pattern Recognit..

[99]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[100]  Jun Yu,et al.  Spontaneous facial micro-expression detection based on deep learning , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[101]  John See,et al.  Eulerian emotion magnification for subtle expression recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[102]  Qi Wu,et al.  For micro-expression recognition: Database and suggestions , 2014, Neurocomputing.

[103]  Xin Geng,et al.  A Relaxed K-SVD Algorithm for Spontaneous Micro-Expression Recognition , 2016, PRICAI.

[104]  Jian Sun,et al.  Joint Cascade Face Detection and Alignment , 2014, ECCV.

[105]  KokSheik Wong,et al.  Optical strain based recognition of subtle emotions , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[106]  Sujing Wang,et al.  A main directional maximal difference analysis for spotting facial movements from long-term videos , 2017, Neurocomputing.

[107]  Stefanos Zafeiriou,et al.  Robust and efficient parametric face alignment , 2011, 2011 International Conference on Computer Vision.

[108]  Hua Lu,et al.  Micro-expression detection using integral projections , 2017 .

[109]  Matti Pietikäinen,et al.  Encoding Local Binary Patterns using the re-parametrization of the second order Gaussian jet , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[110]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[111]  Yantao Tian,et al.  Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[112]  Xiaolan Fu,et al.  Face Recognition and Micro-expression Recognition Based on Discriminant Tensor Subspace Analysis Plus Extreme Learning Machine , 2014, Neural Processing Letters.

[113]  Hao Zheng.  Micro-Expression Recognition based on 2D Gabor Filter and Sparse Representation , 2017 .

[114]  Gregory D. Hager,et al.  Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions , 2009, CVPR.

[115]  S. Porter,et al.  Reading Between the Lies , 2008, Psychological science.

[116]  Xiaolan Fu,et al.  Emotional Context Influences Micro-Expression Recognition , 2014, PloS one.

[117]  Rachel Meyer.  Preprocessing and Descriptor Features for Facial Micro-Expression Recognition , 2015 .

[118]  Xiong Xiong,et al.  Linearly Estimating All Parameters of Affine Motion Using Radon Transform , 2014, IEEE Transactions on Image Processing.

[119]  Junjie Yan,et al.  Face detection by structural models , 2014, Image Vis. Comput..

[120]  John See,et al.  Effective recognition of facial micro-expressions with video motion magnification , 2016, Multimedia Tools and Applications.