Automatic recognition of micro-expressions using local binary patterns on three orthogonal planes and extreme learning machine

The use of micro expressions as a means to understand ones state of mind has received major interest owing to the rapid increase in security threats. The subtle changes that occur on ones face reveals one's hidden intentions. Recognition of these subtle intentions by humans can be challenging as this needs well trained people and is always a time consuming task. Automatic recognition of micro expressions thus promises an avenue to save time and resources. In this paper we propose a framework for detecting the presence of micro-expressions using local binary patterns on three orthogonal planes (LBP-TOP) because of its ability to extract temporal features and extreme learning machine (ELM) because of its fast learning speed. To evaluate the performance of the algorithm, CASME II micro-expression database was used for the experiment. We obtained an accuracy of 96.12% which is a significant improvement when compared with the state-of-the-art methods.

[1]  Xiaolan Fu,et al.  Face Recognition and Micro-expression Recognition Based on Discriminant Tensor Subspace Analysis Plus Extreme Learning Machine , 2014, Neural Processing Letters.

[2]  P. Ekman,et al.  Nonverbal Leakage and Clues to Deception †. , 1969, Psychiatry.

[3]  Ana Belén Moreno,et al.  Differential optical flow applied to automatic facial expression recognition , 2011, Neurocomputing.

[4]  Guoying Zhao,et al.  Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features , 2014, ECCV Workshops.

[5]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Alessandro Vinciarelli,et al.  Canal9: A database of political debates for analysis of social interactions , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[7]  Javier R. Movellan,et al.  The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions , 2014, IEEE Transactions on Affective Computing.

[8]  Matthew L. Jensen,et al.  Detecting Concealment of Intent in Transportation Screening: A Proof of Concept , 2009, IEEE Transactions on Intelligent Transportation Systems.

[9]  Feilong Cao,et al.  A study on effectiveness of extreme learning machine , 2011, Neurocomputing.

[10]  John See,et al.  Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition , 2015, PloS one.

[11]  Marian Stewart Bartlett,et al.  Automatic facial expression recognition for intelligent tutoring systems , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Nicholas Costen,et al.  Micro-Facial Movements: An Investigation on Spatio-Temporal Descriptors , 2014, ECCV Workshops.

[13]  Matti Pietikäinen,et al.  A Spontaneous Micro-expression Database: Inducement, collection and baseline , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[14]  Matti Pietikäinen,et al.  Recognising spontaneous facial micro-expressions , 2011, 2011 International Conference on Computer Vision.

[15]  Qi Wu,et al.  CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[16]  Sridhar Godavarthy,et al.  Microexpression spotting in video using optical strain , 2010 .

[17]  Jarmila Pavlovicova,et al.  Optimization of LBP parameters , 2014, Proceedings ELMAR-2014.

[18]  Qi Wu,et al.  For micro-expression recognition: Database and suggestions , 2014, Neurocomputing.

[19]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[20]  Yantao Tian,et al.  Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[21]  Timothy F. Cootes,et al.  Active Shape Models - 'smart snakes' , 1992, BMVC.

[22]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[23]  Timothy F. Cootes,et al.  Feature Detection and Tracking with Constrained Local Models , 2006, BMVC.

[24]  KokSheik Wong,et al.  Less is More: Micro-expression Recognition from Video using Apex Frame , 2016, Signal Process. Image Commun..

[25]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[26]  S. Weinberger.  Airport security: Intent to deceive? , 2010, Nature.

[27]  Radhika M. Pai,et al.  Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences , 2016, 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI).

[28]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Dmitry B. Goldgof,et al.  Macro- and micro-expression spotting in long videos using spatio-temporal strain , 2011, Face and Gesture 2011.

[30]  Feng Xu,et al.  Microexpression Identification and Categorization Using a Facial Dynamics Map , 2017, IEEE Transactions on Affective Computing.

[31]  Dmitry B. Goldgof,et al.  Automatic expression spotting in videos , 2014, Image Vis. Comput..

[32]  E. A. Haggard,et al.  Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy , 1966 .

[33]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[34]  Min Chen,et al.  Facial expression recognition in dynamic sequences: An integrated approach , 2014, Pattern Recognit..

[35]  Ming Yu,et al.  Micro-expression recognition based on CBP-TOP feature with ELM , 2015 .

[36]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[37]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[38]  Lin Su,et al.  High-Stakes Deception Detection Based on Facial Expressions , 2014, 2014 22nd International Conference on Pattern Recognition.

[39]  Guoying Zhao,et al.  Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space , 2014, 2014 22nd International Conference on Pattern Recognition.

[40]  Matti Pietikäinen,et al.  Local Binary Patterns , 2010, Scholarpedia.

[41]  David J. Kriegman,et al.  The yale face database , 1997 .

[42]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[44]  Yuichi Ohta,et al.  Detection and measurement of facial micro-expression characteristics for psychological analysis (ヒューマン情報処理) , 2010 .

[45]  KokSheik Wong,et al.  Automatic apex frame spotting in micro-expression database , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[46]  Wen-Jing Yan,et al.  How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions , 2013 .

[47]  Matti Pietikäinen,et al.  Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[48]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[49]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[50]  Shiqing Zhang,et al.  A Review on Facial Expression Recognition: Feature Extraction and Classification , 2016 .

[51]  Guoying Zhao,et al.  CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation , 2014, PloS one.

[52]  Takeo Kanade,et al.  Subtly different facial expression recognition and expression intensity estimation , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[53]  M. Phillips,et al.  A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. , 2006, The British journal of clinical psychology.

[54]  Xibin Jia,et al.  Dynamic Feature Extraction for Facial Expression Recognition Based on Optical Flow , 2013 .

[55]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  Gwen Littlewort,et al.  Discrimination of Moderate and Acute Drowsiness Based on Spontaneous Facial Expressions , 2010, 2010 20th International Conference on Pattern Recognition.

[57]  L. Fleischer.  Telling Lies Clues To Deceit In The Marketplace Politics And Marriage , 2016 .

[58]  Yuichi Ohta,et al.  Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor , 2009, ICDP.

[59]  Rouslan A. Moro,et al.  Support Vector Machines (SVM) as a Technique for Solvency Analysis , 2008 .

[60]  Peter Bull,et al.  Detecting Deception from Emotional and Unemotional Cues , 2009 .

[61]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[62]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[63]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[65]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[66]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[67]  Qi Wu,et al.  The Machine Knows What You Are Hiding: An Automatic Micro-expression Recognition System , 2011, ACII.

[68]  L. D. Pool,et al.  Improving emotional intelligence and emotional self efficacy through a teaching intervention for university students. , 2012 .