Rendering Realistic Subject-Dependent Expression Images by Learning 3DMM Deformation Coefficients

Automatic analysis of facial expressions is now attracting an increasing interest, thanks to the many potential applications it can enable. However, collecting images with labeled expression for large sets of images or videos is a quite complicated operation that, in most of the cases, requires substantial human intervention. In this paper, we propose a solution that, starting from a neutral image of a subject, is capable of producing a realistic expressive face image of the same subject. This is possible thanks to the use of a particular 3D morphable model (3DMM) that can effectively and efficiently fit to 2D images, and then deform itself under the action of deformation parameters learned expression-by-expression in a subject-independent manner. Ultimately, the application of such deformation parameters to the neutral model of a subject allows the rendering of realistic expressive images of the subject. Experiments demonstrate that such deformation parameters can be learned from a small set of training data using simple statistical tools; despite this simplicity, very realistic subject-dependent expression renderings can be obtained. Furthermore, robustness to cross dataset tests is also evidenced.

[1]  Adrian Hilton,et al.  A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling , 2011, 2011 International Conference on Computer Vision.

[2]  Thomas Gerig,et al.  Gaussian Process Morphable Models , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[4]  Sami Romdhani,et al.  Optimal Step Nonrigid ICP Algorithms for Surface Registration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Yuchi Huang,et al.  DyadGAN: Generating Facial Expressions in Dyadic Interactions , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[7]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[8]  George Trigeorgis,et al.  3D Face Morphable Models "In-the-Wild" , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ioannis A. Kakadiaris,et al.  End-to-End 3D Face Reconstruction with Deep Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Thomas Vetter,et al.  Expression invariant 3D face recognition with a Morphable Model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[11]  William J. Christmas,et al.  3D Face Tracking and Texture Fusion in the Wild , 2016, ArXiv.

[12]  Ramakant Nevatia,et al.  ExpNet: Landmark-Free, Deep, 3D Facial Expressions , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[13]  Subramanian Ramanathan,et al.  Human Facial Expression Recognition using a 3D Morphable Model , 2006, 2006 International Conference on Image Processing.

[14]  Alan Brunton,et al.  Multilinear Wavelets: A Statistical Shape Space for Human Faces , 2014, ECCV.

[15]  Stefanos Zafeiriou,et al.  A 3D Morphable Model Learnt from 10,000 Faces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Stan Z. Li,et al.  Towards Pose Robust Face Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[18]  Minhua Ma,et al.  A Review of Serious Games for Children with Autism Spectrum Disorders (ASD) , 2014, SGDA.

[19]  Adrian Hilton,et al.  Perceived Emotionality of Linear and Non-Linear AUs Synthesised using a 3D Dynamic Morphable Facial Model , 2015, AVSP.

[20]  Sami Romdhani,et al.  Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[21]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[22]  Hui Chen,et al.  Geometry-Contrastive Generative Adversarial Network for Facial Expression Synthesis , 2018, ArXiv.

[23]  William A. P. Smith,et al.  3D morphable face models revisited , 2009, CVPR.

[24]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Eva Krumhuber,et al.  Perception of linear and nonlinear motion properties using a FACS validated 3D facial model , 2010, APGV '10.

[26]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[27]  Walid Barhoumi,et al.  Fully Automated Facial Expression Recognition Using 3D Morphable Model and Mesh-Local Binary Pattern , 2017, ACIVS.

[28]  Alberto Del Bimbo,et al.  A Dictionary Learning-Based 3D Morphable Shape Model , 2017, IEEE Transactions on Multimedia.

[29]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Michael Spann,et al.  Facial Expression Recognition Using FAPs-Based 3DMMM , 2013 .

[31]  Fotis Liarokapis,et al.  Comparing interaction techniques for serious games through brain-computer interfaces: A user perception evaluation study , 2014, Entertain. Comput..

[32]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[33]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Alberto Del Bimbo,et al.  Pose Independent Face Recognition by Localizing Local Binary Patterns via Deformation Components , 2014, 2014 22nd International Conference on Pattern Recognition.

[35]  Patrik Huber,et al.  A 3D Face Modelling Approach for Pose-Invariant Face Recognition in a Human-Robot Environment , 2016, RoboCup.

[36]  Chi-Ho Chan,et al.  Face Recognition Using a Unified 3D Morphable Model , 2016, ECCV.

[37]  Tal Hassner,et al.  Regressing Robust and Discriminative 3D Morphable Models with a Very Deep Neural Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Thomas Grechenig,et al.  Chances for serious games in rehabilitation of stroke patients on the example of utilizing the Wii Fit Balance Board , 2013, 2013 IEEE 2nd International Conference on Serious Games and Applications for Health (SeGAH).