Recognizing Subtle Micro-facial Expressions Using Fuzzy Histogram of Optical Flow Orientations and Feature Selection Methods

Micro-expressions are the subtle and short-lived facial deformations that convey the inner feelings of a person. Automatic recognition of micro-expressions has potential applications in many areas. However, extraction of the appropriate feature, for encoding the subtle movements during the micro-expressions, is a very challenging work. The use of spatial and spatio-temporal features are studied extensively for this problem. However, the face appearance does not change appreciably during a micro-expression. Moreover, the muscle movements are also very small, almost indistinguishable. Rather, these changes possess a temporal pattern. We use the fuzzy histogram of optical flow orientation (FHOFO) features to encode the temporal patterns associated with facial micro-movements. The FHOFO constructs fuzzified angular histograms from the facial movement vectors. The feature descriptors of a micro-expression clip usually possess high dimension and suffer from the curse of dimensionality. To this end, we explore different feature selection methods to reduce the dimension of the descriptor. Experimentally we found that FHOFO achieves significant accuracy on the publicly available databases and its performance is consistently well across the databases.

[1]  John See,et al.  Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions , 2014, ACCV.

[2]  Osamu Yoshie,et al.  Micro-expression recognition by feature points tracking , 2014, 2014 10th International Conference on Communications (COMM).

[3]  Wen-Jing Yan,et al.  How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions , 2013 .

[4]  Matti Pietikäinen,et al.  Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[5]  Guoying Zhao,et al.  Sparse tensor canonical correlation analysis for micro-expression recognition , 2016, Neurocomputing.

[6]  Dmitry B. Goldgof,et al.  Automatic expression spotting in videos , 2014, Image Vis. Comput..

[7]  Ming Yu,et al.  Micro-expression recognition based on CBP-TOP feature with ELM , 2015 .

[8]  Guoying Zhao,et al.  Spontaneous micro-expression spotting via geometric deformation modeling , 2016, Comput. Vis. Image Underst..

[9]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[10]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[11]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Paul Ekman,et al.  Lie Catching and Microexpressions , 2009 .

[13]  M. Owayjan,et al.  The design and development of a Lie Detection System using facial micro-expressions , 2012, 2012 2nd International Conference on Advances in Computational Tools for Engineering Applications (ACTEA).

[14]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[15]  Matti Pietikäinen,et al.  Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns , 2016, Neurocomputing.

[16]  Xuehua Wang,et al.  Feature selection for high-dimensional imbalanced data , 2013, Neurocomputing.

[17]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[18]  Takeo Kanade,et al.  Facial Expression Recognition , 2011, Handbook of Face Recognition.

[19]  Maja Pantic,et al.  Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics , 2007, ICCV-HCI.

[20]  Matti Pietikäinen,et al.  Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods , 2015, IEEE Transactions on Affective Computing.

[21]  Paul S. Bradley,et al.  Feature Selection via Concave Minimization and Support Vector Machines , 1998, ICML.

[22]  Aurobinda Routray,et al.  An effective feature selection method based on pair-wise feature proximity for high dimensional low sample size data , 2017, 2017 25th European Signal Processing Conference (EUSIPCO).

[23]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[24]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[25]  P. Ekman.  Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life , 2003 .

[26]  Andrea Cavallaro,et al.  Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Matti Pietikäinen,et al.  Spatiotemporal Integration of Optical Flow Vectors for Micro-expression Detection , 2015, ACIVS.

[28]  KokSheik Wong,et al.  Spontaneous Subtle Expression Detection and Recognition based on Facial Strain , 2016, Signal Process. Image Commun..

[29]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[30]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[31]  Guoying Zhao,et al.  Selective deep features for micro-expression recognition , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[32]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[33]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[34]  John See,et al.  Monogenic Riesz wavelet representation for micro-expression recognition , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[35]  Stefanos Zafeiriou,et al.  From Pixels to Response Maps: Discriminative Image Filtering for Face Alignment in the Wild , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Aurobinda Routray,et al.  Automatic facial expression recognition using features of salient facial patches , 2015, IEEE Transactions on Affective Computing.

[37]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[38]  KokSheik Wong,et al.  Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex , 2016, ACCV Workshops.

[39]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[40]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[41]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Aurobinda Routray,et al.  Fuzzy Histogram of Optical Flow Orientations for Micro-Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[43]  Lei Wang,et al.  Efficient Spectral Feature Selection with Minimum Redundancy , 2010, AAAI.

[44]  Yuichi Ohta,et al.  Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS) , 2013, IEICE Trans. Inf. Syst..

[45]  Xiaolan Fu,et al.  Face Recognition and Micro-expression Recognition Based on Discriminant Tensor Subspace Analysis Plus Extreme Learning Machine , 2014, Neural Processing Letters.

[46]  Kai-Kuang Ma,et al.  Fuzzy color histogram and its use in color image retrieval , 2002, IEEE Trans. Image Process..

[47]  Hao Zheng.  Micro-Expression Recognition based on 2D Gabor Filter and Sparse Representation , 2017 .

[48]  Gregory D. Hager,et al.  Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions , 2009, CVPR.

[49]  Tardi Tjahjadi,et al.  A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences , 2015, Pattern Recognit..

[50]  Sujing Wang,et al.  A main directional maximal difference analysis for spotting facial movements from long-term videos , 2017, Neurocomputing.

[51]  Maja Pantic,et al.  A Dynamic Appearance Descriptor Approach to Facial Actions Temporal Modeling , 2014, IEEE Transactions on Cybernetics.

[52]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[53]  Wei-Shi Zheng,et al.  Multi-task mid-level feature learning for micro-expression recognition , 2017, Pattern Recognit..

[54]  Moi Hoon Yap,et al.  Micro-Facial Movement Detection Using Individualised Baselines and Histogram-Based Descriptors , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[55]  Xin Zhang,et al.  Learning effective binary descriptors for micro-expression recognition transferred by macro-information , 2017, Pattern Recognit. Lett..

[56]  Matti Pietikäinen,et al.  Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition , 2012, IEEE Signal Processing Letters.

[57]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[58]  John See,et al.  Eulerian emotion magnification for subtle expression recognition , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[59]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[60]  Qi Wu,et al.  For micro-expression recognition: Database and suggestions , 2014, Neurocomputing.

[61]  Yong Man Ro,et al.  Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition , 2019, IEEE Transactions on Affective Computing.

[62]  Kazuhiko Sumi,et al.  Micro-Expression Recognition for Detecting Human Emotional Changes , 2016, HCI.