Automatic facial expression recognition based on spatiotemporal descriptors

Facial expression's machine analysis is one of the most challenging problems in Human-Computer Interaction (HCI). Naturally, facial expressions depend on subtle movements of facial muscles to show emotional states. After having studied the relations between basic expressions and corresponding facial deformation models, we propose two new textons, VTB and moments on spatiotemporal plane, to describe the transformation of human face during facial expressions. These descriptors aim at catching both general shape changes and motion texture details. Therefore, modeling the temporal behavior of facial expression captures the dynamic deformation of facial components. Finally, SVM based system is used to efficiently recognize the expression for a single image in sequence. Then, the probabilities of all the frames are used to predict the class of the current sequence. The experimental results are evaluated on both Cohan-Kanade and MMI databases. By comparison to other methods, the effectiveness of our method is clearly demonstrated.

[1]  Thomas S. Huang,et al.  A novel approach to expression recognition from non-frontal face images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[2]  Ioannis Pitas,et al.  Texture and shape information fusion for facial expression and facial action unit recognition , 2008, Pattern Recognit..

[3]  Qingshan Liu,et al.  RankBoost with l1 regularization for facial expression recognition and intensity estimation , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[4]  Shang-Hong Lai,et al.  Learning partially-observed hidden conditional random fields for facial expression recognition , 2009, CVPR.

[5]  Nicu Sebe,et al.  Learning Bayesian network classifiers for facial expression recognition both labeled and unlabeled data , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[7]  Mohammed Yeasin,et al.  From facial expression to level of interest: a spatio-temporal approach , 2004, CVPR 2004.

[8]  Ashraf A. Kassim,et al.  A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA , 2009, Pattern Recognit. Lett..

[9]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[10]  Myung Jin Chung,et al.  Extension of cascaded simple feature based face detection to facial expression recognition , 2008, Pattern Recognit. Lett..

[11]  Siu-Yeung Cho,et al.  Expression recognition using fuzzy spatio-temporal modeling , 2008, Pattern Recognit..

[12]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[13]  Bernhard Schölkopf,et al.  Face Detection - Efficient and Rank Deficient , 2004, NIPS.

[14]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[15]  Lifeng Shang,et al.  Nonparametric discriminant HMM and application to facial expression recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[17]  Ioannis Pitas,et al.  A model-based facial expression recognition algorithm using Principal Components Analysis , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[18]  Qingshan Liu,et al.  Boosting encoded dynamic features for facial expression recognition , 2009, Pattern Recognit. Lett..

[19]  M. Bartlett,et al.  Machine Analysis of Facial Expressions , 2007 .

[20]  Markus Flierl,et al.  Facial expression recognition based on graph-preserving sparse non-negative matrix factorization , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[21]  Enrique Muñoz,et al.  Recognising facial expressions in video sequences , 2007, Pattern Analysis and Applications.

[22]  Gwen Littlewort,et al.  Automatic coding of facial expressions displayed during posed and genuine pain , 2009, Image Vis. Comput..

[23]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[24]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[25]  Kin-Man Lam,et al.  Facial expression recognition based on shape and texture , 2009, Pattern Recognit..

[26]  Sungsoo Park,et al.  Subtle facial expression recognition using motion magnification , 2009, Pattern Recognit. Lett..

[27]  Richard Bowden,et al.  The Effect of Pose on Facial Expression Recognition , 2009, BMVC.

[28]  Surendra Ranganath,et al.  Cloud basis function neural network: A modified RBF network architecture for holistic facial expression recognition , 2008, Pattern Recognit..

[29]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Qiang Ji,et al.  A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Shaogang Gong,et al.  Dynamic Facial Expression Recognition Using A Bayesian Temporal Manifold Model , 2006, BMVC.

[32]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[33]  Chi Chung Ko,et al.  Using moment invariants and HMM in facial expression recognition , 2000, 4th IEEE Southwest Symposium on Image Analysis and Interpretation.

[34]  Anil K. Jain,et al.  Handbook of Face Recognition, 2nd Edition , 2011 .

[35]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Yi Ji,et al.  Learning from essential facial parts and local features for automatic facial expression recognition , 2010, 2010 International Workshop on Content Based Multimedia Indexing (CBMI).

[37]  Matti Pietikäinen,et al.  Boosted multi-resolution spatiotemporal descriptors for facial expression recognition , 2009, Pattern Recognit. Lett..

[38]  Yoichi Sato,et al.  Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates , 2007, International Journal of Computer Vision.