3-D Facial Landmark Localization With Asymmetry Patterns and Shape Regression from Incomplete Local Features

We present a method for the automatic localization of facial landmarks that integrates nonrigid deformation with the ability to handle missing points. The algorithm generates sets of candidate locations from feature detectors and performs combinatorial search constrained by a flexible shape model. A key assumption of our approach is that for some landmarks there might not be an accurate candidate in the input set. This is tackled by detecting partial subsets of landmarks and inferring those that are missing, so that the probability of the flexible model is maximized. The ability of the model to work with incomplete information makes it possible to limit the number of candidates that need to be retained, drastically reducing the number of combinations to be tested with respect to the alternative of trying to always detect the complete set of landmarks. We demonstrate the accuracy of the proposed method in the face recognition grand challenge database, where we obtain average errors of approximately 3.5 mm when targeting 14 prominent facial landmarks. For the majority of these our method produces the most accurate results reported to date in this database. Handling of occlusions and surfaces with missing parts is demonstrated with tests on the Bosphorus database, where we achieve an overall error of 4.81 and 4.25 mm for data with and without occlusions, respectively. To investigate potential limits in the accuracy that could be reached, we also report experiments on a database of 144 facial scans acquired in the context of clinical research, with manual annotations performed by experts, where we obtain an overall error of 2.3 mm, with averages per landmark below 3.4 mm for all 14 targeted points and within 2 mm for half of them. The coordinates of automatically located landmarks are made available on-line.

[1]  Luc Van Gool,et al.  Random Forests for Real Time 3D Face Analysis , 2012, International Journal of Computer Vision.

[2]  Ioannis A. Kakadiaris,et al.  3D Facial Landmark Detection under Large Yaw and Expression Variations , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  D. Rueckert,et al.  Predicting the shapes of bones at a joint: application to the shoulder , 2008, Computer methods in biomechanics and biomedical engineering.

[4]  Gábor Székely,et al.  Estimating the Confidence of Statistical Model Based Shape Prediction , 2009, IPMI.

[5]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[6]  Mohammed Bennamoun,et al.  A training-free nose tip detection method from face range images , 2011, Pattern Recognit..

[7]  Cristina Conde,et al.  Automatic 3D Face Feature Points Extraction with Spin Images , 2006, ICIAR.

[8]  Ioannis A. Kakadiaris,et al.  Minimizing Illumination Differences for 3D to 2D Face Recognition Using Lighting Maps , 2014, IEEE Transactions on Cybernetics.

[9]  Federica Marcolin,et al.  3D human face description: landmarks measures and geometrical features , 2012, Image Vis. Comput..

[10]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[11]  Ioannis A. Kakadiaris,et al.  Using Facial Symmetry to Handle Pose Variations in Real-World 3D Face Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Raimondo Schettini,et al.  3D face detection using curvature analysis , 2006, Pattern Recognit..

[13]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  SrivastavaAnuj,et al.  3D Face Recognition under Expressions, Occlusions, and Pose Variations , 2013 .

[15]  Timothy F. Cootes,et al.  Statistical models of appearance for computer vision , 1999 .

[16]  Berk Gökberk,et al.  Regional Registration for Expression Resistant 3-D Face Recognition , 2010, IEEE Transactions on Information Forensics and Security.

[17]  Lijun Yin,et al.  Automatic pose estimation of 3D facial models , 2008, 2008 19th International Conference on Pattern Recognition.

[18]  Jim Austin,et al.  A Machine-Learning Approach to Keypoint Detection and Landmarking on 3D Meshes , 2012, International Journal of Computer Vision.

[19]  Mineichi Kudo,et al.  Comparison of algorithms that select features for pattern classifiers , 2000, Pattern Recognit..

[20]  Alan C. Bovik,et al.  Anthropometric 3D Face Recognition , 2010, International Journal of Computer Vision.

[21]  Daniel Rueckert,et al.  Hierarchical Statistical Shape Analysis and Prediction of Sub-cortical Brain Structures , 2007 .

[22]  Mohammed Bennamoun,et al.  An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Liming Chen,et al.  A coarse-to-fine curvature analysis-based rotation invariant 3D face landmarking , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[24]  Anil K. Jain,et al.  Multimodal Facial Feature Extraction for Automatic 3D Face Recognition , 2005 .

[25]  Paul F. Whelan,et al.  Comparing 3D Descriptors for Local Search of Craniofacial Landmarks , 2012, ISVC.

[26]  Su Ruan,et al.  Statistical Shape Model-based Segmentation of brain MRI Images , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  L. Akarun,et al.  3D Facial Landmarking under Expression, Pose, and Occlusion Variations , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[28]  P. Rousseeuw Least Median of Squares Regression , 1984 .

[29]  Ramesh C. Jain,et al.  Segmentation through Variable-Order Surface Fitting , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Jitendra Malik,et al.  Recognizing Objects in Range Data Using Regional Point Descriptors , 2004, ECCV.

[31]  Patrick J. Flynn,et al.  Rotated Profile Signatures for robust 3D feature detection , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[32]  Paul F. Whelan,et al.  Rotationally Invariant 3D Shape Contexts using Asymmetry Patterns , 2016, GRAPP/IVAPP.

[33]  A. Bovik,et al.  Automated Facial Feature Detection from Portrait and Range Images , 2008, 2008 IEEE Southwest Symposium on Image Analysis and Interpretation.

[34]  Ioannis A. Kakadiaris,et al.  Accurate Landmarking of Three-Dimensional Facial Data in the Presence of Facial Expressions and Occlusions Using a Three-Dimensional Statistical Facial Feature Model , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Nick Pears,et al.  Landmark Localisation in 3D Face Data , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[36]  Marleen de Bruijne,et al.  Quantitative vertebral morphometry using neighbor-conditional shape models , 2007, Medical Image Anal..

[37]  Gábor Székely,et al.  Model-Based Initialisation for Segmentation , 2000, ECCV.

[38]  Andrea Cavallaro,et al.  3-D Face Detection, Landmark Localization, and Registration Using a Point Distribution Model , 2009, IEEE Transactions on Multimedia.

[39]  Paul F. Whelan,et al.  Compensating inaccurate annotations to train 3D facial landmark localization models , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[40]  Robin J. Hennessy,et al.  Frontonasal dysmorphology in bipolar disorder by 3D laser surface imaging and geometric morphometrics: Comparisons with schizophrenia , 2010, Schizophrenia Research.

[41]  NairPrathap,et al.  3-D face detection, landmark localization, and registration using a point distribution model , 2009 .

[42]  Andrew E. Johnson,et al.  Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Chitra Dorai,et al.  COSMOS - A Representation Scheme for 3D Free-Form Objects , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Alexei I. Zhurov,et al.  A Geometric Morphometric Approach to the Analysis of Lip Shape during Speech: Development of a Clinical Outcome Measure , 2013, PLoS ONE.

[45]  Tsz-Ho Yu,et al.  A Novel Genetic Algorithm for 3D Facial Landmark Localization , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[46]  Tania S. Douglas,et al.  Design, Construction, and Testing of a Stereo-Photogrammetric Tool for the Diagnosis of Fetal Alcohol Syndrome in Infants , 2009, IEEE Transactions on Medical Imaging.

[47]  L. Dagum,et al.  OpenMP: an industry standard API for shared-memory programming , 1998 .

[48]  Bülent Sankur,et al.  A comparative study of face landmarking techniques , 2013, EURASIP J. Image Video Process..

[49]  Robin J. Hennessy,et al.  3D laser surface scanning and geometric morphometric analysis of craniofacial shape as an index of cerebro-craniofacial morphogenesis: initial application to sexual dimorphism , 2002, Biological Psychiatry.

[50]  B. Dorizzi,et al.  Precise Localization of Landmarks on 3D Faces using Gabor Wavelets , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[51]  K. Moos,et al.  How accurate is model planning for orthognathic surgery? , 2008, International journal of oral and maxillofacial surgery.

[52]  Paul F. Whelan,et al.  3D Facial Landmark Localization Using Combinatorial Search and Shape Regression , 2012, ECCV Workshops.

[53]  Conrad Sanderson,et al.  Armadillo: An Open Source C++ Linear Algebra Library for Fast Prototyping and Computationally Intensive Experiments , 2010 .

[54]  Emmanuel Dellandréa,et al.  Precise 2.5D facial landmarking via an analysis by synthesis approach , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[55]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[56]  Thomas Vetter,et al.  Probabilistic Modeling and Visualization of the Flexibility in Morphable Models , 2009, IMA Conference on the Mathematics of Surfaces.

[57]  Peter Hammond,et al.  Atypical face shape and genomic structural variants in epilepsy , 2012, Brain : a journal of neurology.

[58]  Maurício Pamplona Segundo,et al.  Automatic Face Segmentation and Facial Landmark Detection in Range Images , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[59]  Dinggang Shen,et al.  Predictive modeling of anatomic structures using canonical correlation analysis , 2004, 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821).

[60]  Guang-Zhong Yang,et al.  Optimal Feature Point Selection and Automatic Initialization in Active Shape Model Search , 2008, MICCAI.

[61]  E M Bronkhorst,et al.  Evaluation of reproducibility and reliability of 3D soft tissue analysis using 3D stereophotogrammetry. , 2009, International journal of oral and maxillofacial surgery.

[62]  Hassen Drira,et al.  3D Face Recognition under Expressions, Occlusions, and Pose Variations , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Anil K. Jain,et al.  Automatic feature extraction for multiview 3D face recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[64]  Brent E Larson,et al.  Accuracy and precision of a 3D anthropometric facial analysis with and without landmark labeling before image acquisition. , 2011, The Angle orthodontist.