3-D Face Alignment Using A Convolutional Point-Set Representation

We present a robust method for estimating the facial pose and shape parameters with a Projected Normalized Coordinate Code (PNCC), a derived image where each pixel is annotated with its corresponding vertex position in a generic 3D model. The method relies on a Convolutional Point-set Representation (CPR), a carefully designed feature map that extracts useful information encoded in pairs or triplets of vertices of PNCC. To estimate the shape and pose parameters, different CPRs are designed to represent mutually independent pose and shape parameters. The CPRs are then fed into multiple convolutional networks (with recurrent layers) in a sequential manner based on the transformation order of weak-perspective projection model to predict the target parameters. Reprojection errors are sampled along a predicted direction and the parameters are updated based on the pattern of reprojection errors. This technique boosts the model’s capability to find a local minimum under challenging scenarios. It is also demonstrated that CPRs obtained from different sources can be merged and contribute to outperforming the current state-of-the-art solutions for 3D face alignment. Experiments indicate that the proposed CPRFA (CPR-based Face Alignment) significantly improves 3D alignment accuracy. Compared to 3DDFA, the proposed method reduces the normalized mean error on AFLW2000 and AFLW database by 25.3% and 20.2% respectively.

[1]  William J. Christmas,et al.  Dynamic Attention-Controlled Cascaded Shape Regression Exploiting Training Data Augmentation and Fuzzy-Set Sample Weighting , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Stefanos Zafeiriou,et al.  300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[3]  Yiying Tong,et al.  FaceWarehouse: A 3D Facial Expression Database for Visual Computing , 2014, IEEE Transactions on Visualization and Computer Graphics.

[4]  Feng Liu,et al.  Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Ioannis A. Kakadiaris,et al.  Facial 3D model registration under occlusions with sensiblepoints-based reinforced hypothesis refinement , 2017, 2017 IEEE International Joint Conference on Biometrics (IJCB).

[6]  Ioannis A. Kakadiaris,et al.  GoDP: Globally Optimized Dual Pathway deep network architecture for facial landmark localization in-the-wild , 2018, Image Vis. Comput..

[7]  Ioannis A. Kakadiaris,et al.  Benchmarking 3D Pose Estimation for Face Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[8]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Sami Romdhani,et al.  A 3D Face Model for Pose and Illumination Invariant Face Recognition , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[10]  Ioannis A. Kakadiaris,et al.  3D-2D face recognition with pose and illumination normalization , 2017, Comput. Vis. Image Underst..

[11]  Xiaoming Liu,et al.  Pose-Invariant Face Alignment with a Single CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Fred L. Bookstein,et al.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Iasonas Kokkinos,et al.  DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Ioannis A. Kakadiaris,et al.  GoDP: Globally optimized dual pathway system for facial landmark localization in-the-wild , 2017, ArXiv.

[18]  Georgios Tzimiropoulos,et al.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks) , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Hao Li,et al.  Learning Dense Facial Correspondences in Unconstrained Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  Pietro Perona,et al.  Robust Face Landmark Estimation under Occlusion , 2013, 2013 IEEE International Conference on Computer Vision.

[21]  I. E. Sutherland,et al.  Three-dimensional data input by tablet , 1974, SIGGRAPH 1974.

[22]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[23]  Charless C. Fowlkes,et al.  Using Segmentation to Predict the Absence of Occluded Parts , 2015, BMVC.

[24]  Gang Sun,et al.  A Key Volume Mining Deep Framework for Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Josef Kittler,et al.  Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Daniel E. Crispell,et al.  Pix2Face: Direct 3D Face Model Estimation , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[27]  George Trigeorgis,et al.  Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Marios Savvides,et al.  Faster than Real-Time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Poses , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Ioannis A. Kakadiaris,et al.  Evaluation of a 3 D-aided Pose Invariant 2 D Face Recognition System , 2017 .

[30]  Horst Bischof,et al.  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[31]  Ioannis A. Kakadiaris,et al.  Robust 3D Face Shape Reconstruction from Single Images via Two-Fold Coupled Structure Learning and Off-the-Shelf Landmark Detectors , 2014, BMVC.

[32]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Ioannis A. Kakadiaris,et al.  End-to-End 3D Face Reconstruction with Deep Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[35]  Jian Sun,et al.  Face Alignment by Explicit Shape Regression , 2012, International Journal of Computer Vision.

[36]  Jongmoo Choi,et al.  Pooling Faces: Template Based Face Recognition with Pooled Face Images , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Sami Romdhani,et al.  Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Josef Kittler,et al.  3D Morphable Models as Spatial Transformer Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[39]  Yangang Wang,et al.  Online modeling for realtime facial animation , 2013, ACM Trans. Graph..