Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

[1]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Paul Ekman,et al.  Lie Catching and Microexpressions , 2009 .

[3]  Yuan Yan Tang,et al.  Face Recognition Under Varying Illumination Using Gradientfaces , 2009, IEEE Transactions on Image Processing.

[4]  P. Ekman,et al.  The ability to detect deceit generalizes across different types of high-stake lies. , 1997, Journal of personality and social psychology.

[5]  P. Ekman,et al.  Nonverbal leakage and clues to deception. , 1969, Psychiatry.

[6]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[7]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[8]  Pawel Strumillo,et al.  Eye-blink detection system for human–computer interaction , 2011, Universal Access in the Information Society.

[9]  Yale Song,et al.  Learning a sparse codebook of facial and body microexpressions for emotion recognition , 2013, ICMI '13.

[10]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[11]  Masahide Kaneko,et al.  Facial-component-based bag of words and PHOG descriptor for facial expression recognition , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[12]  Matti Pietikäinen,et al.  Towards a practical lipreading system , 2011, CVPR 2011.

[13]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[14]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[15]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[16]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[17]  Guoying Zhao,et al.  Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space , 2014, 2014 22nd International Conference on Pattern Recognition.

[18]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[19]  Christopher Joseph Pal,et al.  Facial Expression Analysis Based on High Dimensional Binary Features , 2014, ECCV Workshops.

[20]  Peter Bull,et al.  Detecting Deception from Emotional and Unemotional Cues , 2009 .

[21]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[22]  S. Porter,et al.  Reading Between the Lies , 2008, Psychological science.

[23]  Stefanos Zafeiriou,et al.  Incremental Slow Feature Analysis with Indefinite Kernel for Online Temporal Video Segmentation , 2012, ACCV.

[24]  Matti Pietikäinen,et al.  Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis , 2014, 2014 22nd International Conference on Pattern Recognition.

[25]  KokSheik Wong,et al.  Subtle Expression Recognition Using Optical Strain Weighted Features , 2014, ACCV Workshops.

[26]  P. Ekman.  Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life , 2003 .

[27]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[28]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[29]  Nicholas Costen,et al.  Micro-Facial Movements: An Investigation on Spatio-Temporal Descriptors , 2014, ECCV Workshops.

[30]  Rosalind W. Picard.  Affective Computing , 1997 .

[31]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[32]  Deepshikha Tiwari,et al.  Dynamic texture recognition using local binary pattern , 2016 .

[33]  Matti Pietikäinen,et al.  Facial expression recognition from near-infrared videos , 2011, Image Vis. Comput..

[34]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[35]  John See,et al.  LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition , 2014, ACCV.

[36]  Oscar Déniz-Suárez,et al.  Face recognition using Histograms of Oriented Gradients , 2011, Pattern Recognit. Lett..

[37]  E. A. Haggard,et al.  Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy , 1966 .

[38]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[40]  A. Ardeshir Goshtasby,et al.  Image registration by local approximation methods , 1988, Image Vis. Comput..

[41]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[42]  Dmitry B. Goldgof,et al.  Towards macro- and micro-expression spotting in video using strain patterns , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[43]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[44]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[45]  Yuichi Ohta,et al.  Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS) , 2013, IEICE Trans. Inf. Syst..

[46]  Paul Ekman,et al.  A Few Can Catch a Liar , 1999 .

[47]  KokSheik Wong,et al.  Optical strain based recognition of subtle emotions , 2014, 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[48]  Matti Pietikäinen,et al.  Encoding Local Binary Patterns using the re-parametrization of the second order Gaussian jet , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[49]  D. Matsumoto,et al.  Evidence for training the ability to read microexpressions of emotion , 2011 .

[50]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[51]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Osamu Yoshie,et al.  Micro-expression recognition by feature points tracking , 2014, 2014 10th International Conference on Communications (COMM).

[53]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[54]  Guoying Zhao,et al.  Micro-Expression Recognition Using Color Spaces , 2015, IEEE Transactions on Image Processing.

[55]  Wen-Jing Yan,et al.  How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions , 2013 .

[56]  Yuxiao Hu,et al.  Spontaneous Emotional Facial Expression Detection , 2006, J. Multim..