Measuring dynamic micro-expressions via feature extraction methods

Abstract Historically, the assessment of nonverbal behaviors such as facial expressions and gestures have relied heavily on observations made with the naked eye. Previous studies have shown that the manual analysis of micro-expressions can be both arduous and time consuming, and often suffers from a low level of reliability. However, new technology may bring forth new opportunities to refine this type of investigation. This research applied three feature extraction algorithms from the field of computer science to assist in measuring subtle dynamic movements in the human face. The Constraint Local Model (CLM) algorithm was employed to detect the faces and track the feature points; coordinates changing position across the frame indicated local facial activity. Based on these features points, facial regions of interest (ROIs) were identified for further analysis by Local Binary Pattern (LBP) and Optical Flow (OF). The LBP algorithm was used to extract texture information from the ROIs and measure correlations among frames; OF was used to analyze facial actions based on the main direction and distance of movement. The results of the three methods were then compared with manual coding. The proposed methods, especially OF, proved to be both robust and reliable.

[1]  Wen-Jing Yan,et al.  How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions , 2013 .

[2]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[3]  Peng Zhang,et al.  Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation , 2016, Neural Computing and Applications.

[4]  Jeffrey F Cohn,et al.  Facial expression as an indicator of pain in critically ill intubated adults during endotracheal suctioning. , 2013, American journal of critical care : an official publication, American Association of Critical-Care Nurses.

[5]  Stefanos Zafeiriou,et al.  Robust Discriminative Response Map Fitting with Constrained Local Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Ursula Hess,et al.  The cues decoders use in attempting to differentiate emotion-elicited and posed facial expressions , 1994 .

[7]  Simon Lucey,et al.  Deformable model fitting with a mixture of local experts , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[8]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[9]  Guoying Zhao,et al.  A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition , 2016, IEEE Transactions on Affective Computing.

[10]  Eva Krumhuber,et al.  Moving Smiles: The Role of Dynamic Components for the Perception of the Genuineness of Smiles , 2005 .

[11]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[12]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Gwen Littlewort,et al.  The computer expression recognition toolbox (CERT) , 2011, Face and Gesture 2011.

[14]  Timothy F. Cootes,et al.  Feature Detection and Tracking with Constrained Local Models , 2006, BMVC.

[15]  P. Ekman,et al.  The ability to detect deceit generalizes across different types of high-stake lies. , 1997, Journal of personality and social psychology.

[16]  Ulrich Seidl,et al.  Johannes Schröder Symptoms Facial Expression in Alzheimer ' s Disease : Impact of Cognitive Deficits and Neuropsychiatric , 2012 .

[17]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[18]  Marian Stewart Bartlett,et al.  Multilayer Architectures for Facial Action Unit Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[20]  J. Cohn,et al.  Spontaneous facial expression in a small group can be automatically measured: An initial demonstration , 2010, Behavior research methods.

[21]  Venugopal Govindaraju,et al.  Behavior and Security , 2009 .

[22]  J. Cohn,et al.  All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous , 2009, Journal of nonverbal behavior.

[23]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[25]  Rui Yan,et al.  Micro-expression recognition system , 2016 .

[26]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Paul Ekman,et al.  Lie Catching and Microexpressions , 2009 .

[28]  J. Gibson.  The perception of the visual world , 1951 .

[29]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[30]  Arvid Kappas,et al.  Of Butterflies and Roaring Thunder , 2003 .

[31]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[32]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[33]  Nicu Sebe,et al.  Authentic facial expression analysis , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[34]  S. Porter,et al.  Reading Between the Lies , 2008, Psychological science.

[35]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[36]  Maja Pantic,et al.  A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Qi Wu,et al.  For micro-expression recognition: Database and suggestions , 2014, Neurocomputing.

[38]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[39]  Tobias Senst,et al.  Robust local optical flow estimation using bilinear equations for sparse motion estimation , 2013, 2013 IEEE International Conference on Image Processing.

[40]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).