Recognizing Spontaneous Micro-Expression Using a Three-Stream Convolutional Neural Network

Micro-expression recognition (MER) has attracted much attention with various practical applications, particularly in clinical diagnosis and interrogations. In this paper, we propose a three-stream convolutional neural network (TSCNN) to recognize MEs by learning ME-discriminative features in three key frames of ME videos. We design a dynamic-temporal stream, static-spatial stream, and local-spatial stream module for the TSCNN that respectively attempt to learn and integrate temporal, entire facial region, and facial local region cues in ME videos with the goal of recognizing MEs. In addition, to allow the TSCNN to recognize MEs without using the index values of apex frames, we design a reliable apex frame detection algorithm. Extensive experiments are conducted with five public ME databases: CASME II, SMIC-HS, SAMM, CAS(ME)2, and CASME. Our proposed TSCNN is shown to achieve more promising recognition results when compared with many other methods.

[1]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Zheng Lian,et al.  Discriminative Video Representation with Temporal Order for Micro-expression Recognition , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  Dmitry Chetverikov,et al.  Qualitative Characterization of Dynamic Textures for Video Retrieval , 2004, ICCVG.

[5]  Mark G. Frank,et al.  Police Lie Detection Accuracy: The Effect of Lie Scenario , 2009, Law and human behavior.

[6]  Guoying Zhao,et al.  Learning From Hierarchical Spatiotemporal Descriptors for Micro-Expression Recognition , 2018, IEEE Transactions on Multimedia.

[7]  Loong Fah Cheong,et al.  Synergizing spatial and temporal texture , 2002, IEEE Trans. Image Process..

[8]  Huai-Qian Khor,et al.  Enriched Long-Term Recurrent Convolutional Network for Facial Micro-Expression Recognition , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[9]  John See,et al.  Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application , 2016, IEEE Transactions on Affective Computing.

[10]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[11]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[13]  Xiaolan Fu,et al.  CAS(ME)$^2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition , 2018, IEEE Transactions on Affective Computing.

[14]  Randal C. Nelson,et al.  Qualitative recognition of motion using temporal texture , 1992, CVGIP Image Underst..

[15]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Xin Geng,et al.  A Relaxed K-SVD Algorithm for Spontaneous Micro-Expression Recognition , 2016, PRICAI.

[17]  Huai-Qian Khor,et al.  Dual-stream Shallow Networks for Facial Micro-expression Recognition , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[18]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[20]  Patrick Bouthemy,et al.  Motion characterization from temporal cooccurrences of local motion-based measures for video indexing , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[21]  Yan Liu,et al.  Deep residual learning for image steganalysis , 2018, Multimedia Tools and Applications.

[22]  Kidiyo Kpalma,et al.  Motion descriptors for micro-expression recognition , 2018, Signal Process. Image Commun..

[23]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[24]  KokSheik Wong,et al.  Optical strain based recognition of subtle emotions , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[25]  Matti Pietikäinen,et al.  Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods , 2015, IEEE Transactions on Affective Computing.

[26]  Matti Pietikäinen,et al.  Capturing correlations of local features for image representation , 2016, Neurocomputing.

[27]  Radhika M. Pai,et al.  Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences , 2016, 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[28]  Nicholas Costen,et al.  SAMM: A Spontaneous Micro-Facial Movement Dataset , 2018, IEEE Transactions on Affective Computing.

[29]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[32]  Ming Wan,et al.  Combining 3D Convolutional Neural Networks with Transfer Learning by Supervised Pre-Training for Facial Micro-Expression Recognition , 2019, IEICE Trans. Inf. Syst..

[33]  John See,et al.  A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges , 2018, Front. Psychol..

[34]  John See,et al.  Eulerian emotion magnification for subtle expression recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[35]  Paul Ekman,et al.  A Few Can Catch a Liar , 1999 .

[36]  Matti Pietikäinen,et al.  Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[37]  KokSheik Wong,et al.  Micro-expression recognition using apex frame with phase information , 2017, 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[38]  Cha Zhang,et al.  Image based Static Facial Expression Recognition with Multiple Deep Network Learning , 2015, ICMI.

[39]  Matti Pietikäinen,et al.  Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[40]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[41]  Patrick Bouthemy,et al.  Motion Recognition Using Nonparametric Image Motion Models Estimated from Temporal and Multiscale Cooccurrence Statistics , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Snehasis Mukherjee,et al.  Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[43]  Yong Man Ro,et al.  Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations , 2016, ACM Multimedia.

[44]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[46]  Matti Pietikäinen,et al.  Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition , 2012, IEEE Signal Processing Letters.

[47]  Haiping Lu,et al.  MPCA: Multilinear Principal Component Analysis of Tensor Objects , 2008, IEEE Transactions on Neural Networks.

[48]  Khashayar Khorasani,et al.  Facial expression recognition using constructive feedforward neural networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[49]  John See,et al.  Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions , 2014, ACCV.

[50]  Chi-Ho Chan,et al.  Local Ordinal Contrast Pattern Histograms for Spatiotemporal, Lip-Based Speaker Authentication , 2012, IEEE Trans. Inf. Forensics Secur..

[51]  Guoying Zhao,et al.  Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern , 2017, 2017 International Conference on the Frontiers and Advances in Data Science (FADS).

[52]  Yong Man Ro,et al.  Subtle Facial Expression Recognition Using Adaptive Magnification of Discriminative Facial Motion , 2015, ACM Multimedia.

[53]  Min Peng,et al.  Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition , 2017, Front. Psychol..

[54]  Guoying Zhao,et al.  Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions , 2019, IEEE Transactions on Multimedia.

[55]  John See,et al.  Micro-expression recognition based on 3D flow convolutional neural network , 2018, Pattern Analysis and Applications.

[56]  Christopher Joseph Pal,et al.  Facial Expression Analysis Based on High Dimensional Binary Features , 2014, ECCV Workshops.

[57]  Wei-Chuen Yau,et al.  OFF-ApexNet on Micro-expression Recognition System , 2018, Signal Process. Image Commun..

[58]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[59]  John See,et al.  Monogenic Riesz wavelet representation for micro-expression recognition , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[60]  Matti Pietikäinen,et al.  Spontaneous Facial Micro-Expression Recognition using Discriminative Spatiotemporal Local Binary Pattern with an Improved Integral Projection , 2016, ArXiv.

[61]  Takeshi Tokuyama,et al.  CapsuleNet for Micro-Expression Recognition , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[62]  Patrick Bouthemy,et al.  Motion recognition using spatio-temporal random walks in sequence of 2D motion-related measurements , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[63]  Xiaolan Fu,et al.  SMEConvNet: A Convolutional Neural Network for Spotting Spontaneous Facial Micro-Expression From Long Videos , 2018, IEEE Access.

[64]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[65]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[66]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[67]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[68]  Subrahmanyam Murala,et al.  LEARNet: Dynamic Imaging Network for Micro Expression Recognition , 2019, IEEE Transactions on Image Processing.

[69]  Esa Rahtu,et al.  Volume Local Phase Quantization for Blur-Insensitive Dynamic Texture Classification , 2011, SCIA.

[70]  KokSheik Wong,et al.  Less is More: Micro-expression Recognition from Video using Apex Frame , 2016, Signal Process. Image Commun..

[71]  Weixin Xie,et al.  Dynamic Texture Recognition by Spatio-Temporal Multiresolution Histograms , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[72]  Guoying Zhao,et al.  Can Micro-Expression be Recognized Based on Single Apex Frame? , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[73]  Venugopal Govindaraju,et al.  Behavior and Security , 2009 .

[74]  Jun Yu,et al.  Micro-expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow , 2018, 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT).

[75]  Dmitry Chetverikov,et al.  A Brief Survey of Dynamic Texture Description and Recognition , 2005, CORES.

[76]  KokSheik Wong,et al.  Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex , 2016, ACCV Workshops.

[77]  Matti Pietikäinen,et al.  Towards a dynamic expression recognition system under facial occlusion , 2012, Pattern Recognit. Lett..

[78]  John See,et al.  LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition , 2014, ACCV.

[79]  Ling Zhou,et al.  Dual-Inception Network for Cross-Database Micro-Expression Recognition , 2019, 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).

[80]  Dmitry Chetverikov,et al.  Dynamic Texture Recognition Using Normal Flow and Texture Regularity , 2005, IbPRIA.