3D shape estimation in video sequences provides high precision evaluation of facial expressions

Person independent and pose invariant estimations of facial expressions and action unit (AU) intensity estimation are important for situation analysis and for automated video annotation. We evaluated raw 2D shape data of the CK+ database, used Procrustes transformation and the multi-class SVM leave-one-out method for classification. We found close to 100% performance demonstrating the relevance and the strength of details of the shape. Precise 3D shape information was computed by means of constrained local models (CLM) on video sequences. Such sequences offer the opportunity to compute a time-averaged '3D personal mean shape' (PMS) from the estimated CLM shapes, which - upon subtraction - gives rise to person independent emotion estimation. On CK+ data PMS showed significant improvements over AU0 normalization; performance reached and sometimes surpassed state-of-the-art results on emotion classification and on AU intensity estimation. 3D PMS from 3D CLM offers pose invariant emotion estimation that we studied by rendering a 3D emotional database for different poses and different subjects from the BU 4DFE database. Frontal shapes derived from CLM fits of the 3D shape were evaluated. Results demonstrate that shape estimation alone can be used for robust, high quality pose invariant emotion classification and AU intensity estimation.

[1]  Jacob Whitehill,et al.  Haar features for FACS AU recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[2]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[3]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[4]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[5]  Jeffrey F. Cohn,et al.  Painful data: The UNBC-McMaster shoulder pain expression archive database , 2011, Face and Gesture 2011.

[6]  Maja Pantic,et al.  A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Gwen Littlewort,et al.  Fully Automatic Facial Action Recognition in Spontaneous Behavior , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[8]  László A. Jeni,et al.  Efficient, Pose Invariant Facial Emotion Classification using 3D Constrained Local Model and 2D Shape Information , 2011, CVPR 2011.

[9]  Sridha Sridharan,et al.  Person-independent facial expression detection using Constrained Local Models , 2011, Face and Gesture 2011.

[10]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[11]  Stefanos Zafeiriou,et al.  A dynamic approach to the recognition of 3D facial expressions and their temporal models , 2011, Face and Gesture 2011.

[12]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[13]  Thomas S. Huang,et al.  Expression recognition from 3D dynamic faces using robust spatio-temporal shape features , 2011, Face and Gesture 2011.

[14]  Rogério Schmidt Feris,et al.  Manifold Based Analysis of Facial Expression , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[16]  Lijun Yin,et al.  Tracking Vertex Flow and Model Adaptation for Three-Dimensional Spatiotemporal Face Analysis , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  Lijun Yin,et al.  Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[18]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[19]  András Lörincz,et al.  Relating Priming and Repetition Suppression , 2002, Int. J. Neural Syst..

[20]  András Lörincz,et al.  High quality facial expression recognition in video streams using shape related information only , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[21]  Qiang Ji,et al.  Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[23]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[24]  J. Movellan,et al.  Human and computer recognition of facial expressions of emotion , 2007, Neuropsychologia.

[25]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[26]  Lijun Yin,et al.  Facial Expression Recognition Based on 3D Dynamic Range Model Sequences , 2008, ECCV.

[27]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[28]  J. Winn,et al.  Darwin , 1883, Nature.

[29]  David Cristinacce,et al.  Automatic feature localisation with constrained local models , 2008, Pattern Recognit..

[30]  Thomas S. Huang,et al.  Emotion Recognition from Arbitrary View Facial Images , 2010, ECCV.

[31]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[32]  Maja Pantic,et al.  Coupled Gaussian Process Regression for Pose-Invariant Facial Expression Recognition , 2010, ECCV.

[33]  Rama Chellappa,et al.  Towards view-invariant expression analysis using analytic shape manifolds , 2011, Face and Gesture 2011.

[34]  Simon Lucey,et al.  Deformable Model Fitting by Regularized Landmark Mean-Shift , 2010, International Journal of Computer Vision.

[35]  John C. Dalrymple-Alford,et al.  Sensitivity to genuine versus posed emotion specified in facial displays , 2010 .