Recognition of 3D facial expression dynamics

In this paper we propose a method that exploits 3D motion-based features between frames of 3D facial geometry sequences for dynamic facial expression recognition. An expressive sequence is modelled to contain an onset followed by an apex and an offset. Feature selection methods are applied in order to extract features for each of the onset and offset segments of the expression. These features are then used to train GentleBoost classifiers and build a Hidden Markov Model in order to model the full temporal dynamics of the expression. The proposed fully automatic system was employed on the BU-4DFE database for distinguishing between the six universal expressions: Happy, Sad, Angry, Disgust, Surprise and Fear. Comparisons with a similar 2D system based on the motion extracted from facial intensity images was also performed. The attained results suggest that the use of the 3D information does indeed improve the recognition accuracy when compared to the 2D data in a fully automatic manner.

[1]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[2]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[3]  Michael G. Strintzis,et al.  BIlinear Decomposition of 3-D face images: An application to facial expression recognition , 2009, 2009 10th Workshop on Image Analysis for Multimedia Interactive Services.

[4]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[5]  Mohammed Yeasin,et al.  Recognition of facial expressions and measurement of levels of interest from video , 2006, IEEE Transactions on Multimedia.

[6]  Nicu Sebe,et al.  Authentic Facial Expression Analysis , 2004, FGR.

[7]  Bernd Girod,et al.  Model-based face tracking for view-independent facial expression recognition , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[8]  Subramanian Ramanathan,et al.  Human Facial Expression Recognition using a 3D Morphable Model , 2006, 2006 International Conference on Image Processing.

[9]  Thomas S. Huang,et al.  Capturing subtle facial motions in 3D face tracking , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[10]  Mohammed Yeasin,et al.  From facial expression to level of interest: a spatio-temporal approach , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[11]  Maja Pantic,et al.  Machine analysis of facial behaviour: naturalistic and dynamic behaviour , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[12]  Luiz Velho,et al.  Automatic 3D Facial Expression Analysis in Videos , 2005, AMFG.

[13]  Maja Pantic,et al.  Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[14]  Hasan Demirel,et al.  Facial Expression Recognition Using 3D Facial Feature Distances , 2007, ICIAR.

[15]  H. Demirel,et al.  3D facial expression recognition with geometrically localized facial features , 2008, 2008 23rd International Symposium on Computer and Information Sciences.

[16]  Ioannis Pitas,et al.  Discriminant Graph Structures for Facial Expression Recognition , 2008, IEEE Transactions on Multimedia.

[17]  Amanda C.C. Williams,et al.  Facial expression of pain: An evolutionary account , 2002, Behavioral and Brain Sciences.

[18]  Michael G. Strintzis,et al.  Bilinear elastically deformable models with application to 3D face and facial expression recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[19]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Sotiris Malassiotis,et al.  Robust facial action recognition from real-time 3D streams , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[21]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Hatice Gunes,et al.  How to distinguish posed from spontaneous smiles using geometric features , 2007, ICMI '07.

[23]  Ragini Verma,et al.  Quantifying Facial Expression Abnormality in Schizophrenia by Combining 2D and 3D Features , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Maja Pantic,et al.  Detecting facial actions and their temporal segments in nearly frontal-view face image sequences , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[25]  Maja Pantic,et al.  Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics , 2007, ICCV-HCI.

[26]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[27]  Arman Savran,et al.  Facial action unit detection: 3D versus 2D modality , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[28]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[29]  Qiang Ji,et al.  Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Alejandro F Frangi,et al.  Automatic construction of 3-D statistical deformation models of the brain using nonrigid registration , 2003, IEEE Transactions on Medical Imaging.

[31]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[32]  Lijun Yin,et al.  Facial Expression Recognition Based on 3D Dynamic Range Model Sequences , 2008, ECCV.

[33]  Thomas S. Huang,et al.  3D facial expression recognition based on automatically selected features , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[34]  Trevor Hastie,et al.  Additive Logistic Regression : a Statistical , 1998 .

[35]  Lijun Yin,et al.  Analyzing Facial Expressions Using Intensity-Variant 3D Data For Human Computer Interaction , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[36]  Zhengyou Zhang,et al.  Iterative point matching for registration of free-form curves and surfaces , 1994, International Journal of Computer Vision.

[37]  Marco Costa,et al.  Social Presence, Embarrassment, and Nonverbal Behavior , 2001 .

[38]  Qiang Ji,et al.  Active and dynamic information fusion for facial expression understanding from image sequences , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Arman Savran,et al.  Non-rigid registration of 3D surfaces by deformable 2D triangular meshes , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[40]  Sotiris Malassiotis,et al.  Real-time 2D+3D facial action and expression recognition , 2010, Pattern Recognit..

[41]  Nicu Sebe,et al.  Facial expression recognition from video sequences: temporal and static modeling , 2003, Comput. Vis. Image Underst..

[42]  Arman Savran,et al.  Automatic detection of facial actions from 3D data , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[43]  Nicu Sebe,et al.  Facial expression recognition from video sequences , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[44]  Changbo Hu,et al.  Probabilistic expression analysis on manifolds , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[45]  J. Cohn,et al.  Deciphering the Enigmatic Face , 2005, Psychological science.

[46]  Jun Wang,et al.  3D Facial Expression Recognition Based on Primitive Surface Feature Distribution , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[47]  Maja Pantic,et al.  A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, CVPR Workshops.

[49]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[50]  Michael G. Strintzis,et al.  Bilinear Models for 3-D Face and Facial Expression Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[51]  Lisa Gralewski,et al.  Using a tensor framework for the analysis of facial dynamics , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).