A survey of micro-expression recognition

Abstract The limited capacity to recognize micro-expressions with subtle and rapid motion changes is a long-standing problem that presents a unique challenge for expression recognition systems and even for humans. The problem regarding micro-expression is less covered by research when compared to macro-expression. Nevertheless, micro-expression recognition (MER) is imperative to exploit the full potential of expression recognition for real-world applications. Recent MER systems generally focus on three important issues: overfitting caused by a lack of sufficient training data, the imbalanced distribution of samples, and robust features for improving the accuracy of recognition. In this paper, we provide a comprehensive survey on MER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature. We then describe the pre-processing in the standard pipeline of an MER system. For the state of the art in MER, we divide the existing novel algorithms into 6 different tasks according to the type of classes and evaluation protocols. Detailed experiment settings and competitive performances for those 6 tasks are summarized in this section. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust MER systems.

[1]  Michael S. Ryoo,et al.  Representation Flow for Action Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Esa Rahtu,et al.  Volume Local Phase Quantization for Blur-Insensitive Dynamic Texture Classification , 2011, SCIA.

[3]  Ling Zhou,et al.  Dual-Inception Network for Cross-Database Micro-Expression Recognition , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[4]  Chaoyi Zhang,et al.  ICE-GAN: Identity-aware and Capsule-Enhanced GAN for Micro-Expression Recognition and Synthesis , 2020, ArXiv.

[5]  Guoying Zhao,et al.  Learning a Target Sample Re-Generator for Cross-Database Micro-Expression Recognition , 2017, ACM Multimedia.

[6]  Mubarak Shah,et al.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild , 2012, ArXiv.

[7]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[8]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[9]  Nicholas Costen,et al.  Micro-Facial Movements: An Investigation on Spatio-Temporal Descriptors , 2014, ECCV Workshops.

[10]  Guoying Zhao,et al.  Learning From Hierarchical Spatiotemporal Descriptors for Micro-Expression Recognition , 2018, IEEE Transactions on Multimedia.

[11]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[12]  Min Peng,et al.  Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition , 2017, Front. Psychol..

[13]  Matti Pietikäinen,et al.  Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[14]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[15]  Qirong Mao,et al.  Cross-Database Micro-Expression Recognition: A Style Aggregated and Attention Transfer Approach , 2019, 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[16]  Hong-Han Shuai,et al.  MER-GCN: Micro-Expression Recognition Based on Relation Modeling with Graph Convolutional Networks , 2020, 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[17]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[18]  Girdhari Singh,et al.  Non-Linearities Improve OrigiNet based on Active Imaging for Micro Expression Recognition , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[19]  Guoying Zhao,et al.  Cross-Database Micro-Expression Recognition: A Benchmark , 2018, IEEE Transactions on Knowledge and Data Engineering.

[20]  Zeyu Pan,et al.  Eulerian Motion Based 3DCNN Architecture for Facial Micro-Expression Recognition , 2020, MMM.

[21]  Stefanos Zafeiriou,et al.  Robust Discriminative Response Map Fitting with Constrained Local Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Ming Yu,et al.  Spatiotemporal features selection for spontaneous micro-expression recognition , 2018, J. Intell. Fuzzy Syst..

[23]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Huai-Qian Khor,et al.  A Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition System , 2019, ArXiv.

[25]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[27]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[28]  Qiang Wu,et al.  A survey: facial micro-expression recognition , 2017, Multimedia Tools and Applications.

[29]  Matti Pietikäinen,et al.  Facial expression recognition from near-infrared videos , 2011, Image Vis. Comput..

[30]  Nicholas Costen,et al.  SAMM: A Spontaneous Micro-Facial Movement Dataset , 2018, IEEE Transactions on Affective Computing.

[31]  John See,et al.  MEGC 2019 – The Second Facial Micro-Expressions Grand Challenge , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[32]  Timothy F. Cootes,et al.  Feature Detection and Tracking with Constrained Local Models , 2006, BMVC.

[33]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[34]  Matti Pietikäinen,et al.  Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods , 2015, IEEE Transactions on Affective Computing.

[35]  Chuang Gan,et al.  End-to-End Learning of Motion Representation for Video Understanding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Sujing Wang,et al.  Micro-expression recognition with small sample size by transferring long-term convolutional neural network , 2018, Neurocomputing.

[37]  Wei-Shi Zheng,et al.  Multi-task mid-level feature learning for micro-expression recognition , 2017, Pattern Recognit..

[38]  John See,et al.  Are subtle expressions too sparse to recognize? , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[39]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Ioan Marius Bilasco,et al.  Consistent Optical Flow Maps for Full and Micro Facial Expression Recognition , 2017, VISIGRAPP.

[42]  Wei-Chuen Yau,et al.  OFF-ApexNet on Micro-expression Recognition System , 2018, Signal Process. Image Commun..

[43]  Subrahmanyam Murala,et al.  LEARNet: Dynamic Imaging Network for Micro Expression Recognition , 2019, IEEE Transactions on Image Processing.

[44]  Yuning Jiang,et al.  Extensive Facial Landmark Localization with Coarse-to-Fine Convolutional Network Cascade , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[45]  Huicheng Zheng,et al.  A Delaunay-Based Temporal Coding Model for Micro-expression Recognition , 2014, ACCV Workshops.

[46]  KokSheik Wong,et al.  Less is More: Micro-expression Recognition from Video using Apex Frame , 2016, Signal Process. Image Commun..

[47]  Liang Zheng,et al.  A Neural Micro-Expression Recognizer , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[48]  Anastasios Delopoulos,et al.  The MUG facial expression database , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[49]  John See,et al.  A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges , 2018, Front. Psychol..

[50]  Zhaoqiang Xia,et al.  Cross-database Micro-Expression Recognition with Deep Convolutional Networks , 2019, ICBEA.

[51]  John See,et al.  Micro-Expression Motion Magnification: Global Lagrangian vs. Local Eulerian Approaches , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[52]  Huai-Qian Khor,et al.  Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[53]  Takeshi Tokuyama,et al.  CapsuleNet for Micro-Expression Recognition , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[54]  KokSheik Wong,et al.  Micro-expression recognition using apex frame with phase information , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[55]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[56]  John See,et al.  Effective recognition of facial micro-expressions with video motion magnification , 2016, Multimedia Tools and Applications.

[57]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[58]  Guoying Zhao,et al.  Can Micro-Expression be Recognized Based on Single Apex Frame? , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[59]  Li Li Lim,et al.  Micro-expression recognition: an updated review of current trends, challenges and solutions , 2018, The Visual Computer.

[60]  Joachim Denzler,et al.  ImageNet pre-trained models with batch normalization , 2016, ArXiv.

[61]  John See,et al.  LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition , 2014, ACCV.

[62]  John See,et al.  Facial Micro-Expressions Grand Challenge 2018 Summary , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[63]  Guoying Zhao,et al.  Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern , 2017, 2017 International Conference on the Frontiers and Advances in Data Science (FADS).

[64]  Guoying Zhao,et al.  Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space , 2014, 2014 22nd International Conference on Pattern Recognition.

[65]  Snehasis Mukherjee,et al.  Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[66]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[67]  KokSheik Wong,et al.  Optical strain based recognition of subtle emotions , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[68]  Min Peng,et al.  From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[69]  Yuichi Ohta,et al.  Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS) , 2013, IEICE Trans. Inf. Syst..

[70]  Xiaolan Fu,et al.  CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition , 2018, IEEE Transactions on Affective Computing.

[71]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[72]  Dong Liu,et al.  High-Resolution Representations for Labeling Pixels and Regions , 2019, ArXiv.

[73]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[74]  Hubert Konik,et al.  Mean Oriented Riesz Features for Micro Expression Classification , 2020, Pattern Recognit. Lett..

[75]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[76]  Huai-Qian Khor,et al.  Enriched Long-Term Recurrent Convolutional Network for Facial Micro-Expression Recognition , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[77]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[78]  John See,et al.  Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application , 2016, IEEE Transactions on Affective Computing.

[79]  John See,et al.  Eulerian emotion magnification for subtle expression recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[80]  Guoying Zhao,et al.  Sparse tensor canonical correlation analysis for micro-expression recognition , 2016, Neurocomputing.

[81]  Xiaoou Tang,et al.  Facial Landmark Detection by Deep Multi-task Learning , 2014, ECCV.

[82]  P. Ekman.  Telling lies: clues to deceit in the marketplace , 1985 .

[83]  Matti Pietikäinen,et al.  Towards a practical lipreading system , 2011, CVPR 2011.

[84]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[85]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[86]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[87]  John See,et al.  Micro-expression recognition based on 3D flow convolutional neural network , 2018, Pattern Analysis and Applications.

[88]  Kidiyo Kpalma,et al.  Motion descriptors for micro-expression recognition , 2018, Signal Process. Image Commun..

[89]  Tobias Senst,et al.  Robust Local Optical Flow for Feature Tracking , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[90]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[91]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[92]  Matti Pietikäinen,et al.  Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns , 2016, Neurocomputing.

[93]  Matti Pietikäinen,et al.  Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[94]  Huai-Qian Khor,et al.  Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition , 2020, IEEE Transactions on Image Processing.

[95]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[96]  Yong Man Ro,et al.  Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations , 2016, ACM Multimedia.

[97]  Partha Pratim Roy,et al.  Facial Micro-expression Spotting and Recognition Using Time Contrasted Feature with Visual Memory , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[98]  Yong Man Ro,et al.  Subtle Facial Expression Recognition Using Adaptive Magnification of Discriminative Facial Motion , 2015, ACM Multimedia.

[99]  Jiřı́ Matas,et al.  Spotting Facial Micro-Expressions “ In the Wild ” , 2017 .

[100]  Guoying Zhao,et al.  Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions , 2019, IEEE Transactions on Multimedia.

[101]  Moi Hoon Yap,et al.  Objective Classes for Micro-Facial Expression Recognition , 2017, J. Imaging.

[102]  John See,et al.  Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions , 2014, ACCV.

[103]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[104]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[105]  Kaiqi Huang,et al.  Random walk-based feature learning for micro-expression recognition , 2018, Pattern Recognit. Lett..

[106]  Guoying Zhao,et al.  Quantifying Micro-expressions with Constraint Local Model and Local Binary Pattern , 2014, ECCV Workshops.

[107]  Jennifer Annoni,et al.  Sparsity-promoting dynamic mode decomposition for systems with inputs , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[108]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[109]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[110]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[111]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.