meshSIFT: Local surface features for 3D face recognition under expression variations and partial data

Matching 3D faces for recognition is a challenging task caused by the presence of expression variations, missing data, and outliers. In this paper the meshSIFT algorithm and its use for 3D face recognition is presented. This algorithm consists of four major components. First, salient points on the 3D facial surface are detected as mean curvature extrema in scale space. Second, orientations are assigned to each of these salient points. Third, the neighbourhood of each salient point is described in a feature vector consisting of concatenated histograms of shape indices and slant angles. Fourth, the feature vectors of two 3D facial surfaces are reliably matched by comparing the angles in feature space. This results in an algorithm which is robust to expression variations, missing data and outliers. As a first contribution, we demonstrate that the number of matching meshSIFT features is a reliable measure for expression-invariant face recognition, as shown by the rank 1 recognition rate of 93.7% and 89.6% for the Bosphorus and FRGC v2 database, respectively. Next, we demonstrate that symmetrising the feature descriptors allows comparing two 3D facial surfaces with limited or no overlap. Validation on the data of the ''SHREC'11: Face Scans'' contest, containing many partial scans, resulted in a recognition rate of 98.6%, clearly outperforming all other participants in the challenge. Finally, we also demonstrate the use of meshSIFT for two other problems related with 3D face recognition: pose normalisation and symmetry plane estimation. For both problems, applying meshSIFT in combination with RANSAC resulted in a correct solution for +/-90% of all Bosphorus database meshes (except +/-90^o and +/-45^o rotations).

[1]  Evangelos E. Milios,et al.  Matching range images of human faces , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[2]  Vinod Chandran,et al.  3D Face Recognition using Log-Gabor Templates , 2006, BMVC.

[3]  A. Bovik,et al.  Three Dimensional Face Recognition Using Iso-Geodesic and Iso-Depth Curves , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[4]  Abolghasem A. Raie,et al.  2.5D face recognition using Patch Geodesic Moments , 2012, Pattern Recognit..

[5]  Ping Yan,et al.  An Automatic 3D Ear Recognition System , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[6]  Gordon Erlebacher,et al.  A novel technique for face recognition using range imaging , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[7]  Jake K. Aggarwal,et al.  Three dimensional face recognition based on geodesic and Euclidean distances , 2007, Electronic Imaging.

[8]  Hassen Drira,et al.  SHREC '11 Track: 3D Face Models Retrieval , 2011, 3DOR@Eurographics.

[9]  Jing Hua,et al.  Surface matching with salient keypoints in geodesic scale space , 2008 .

[10]  Jim Austin,et al.  Three-Dimensional Face Recognition Using Surface Space Combinations , 2004, BMVC.

[11]  Patrick J. Flynn,et al.  Multiple Nose Region Matching for 3D Face Recognition under Varying Facial Expression , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  James J. Little,et al.  Vision-based mobile robot localization and mapping using scale-invariant features , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[13]  Chris Boehnen,et al.  3D Face Recognition Using 3D Alignment for PCA , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[14]  Alberto Del Bimbo,et al.  SHREC’08 entry: 3D face recognition using integral shape information , 2008, 2008 IEEE International Conference on Shape Modeling and Applications.

[15]  Paul Suetens,et al.  Feature detection on 3D face surfaces for pose normalisation and recognition , 2010, 2010 Fourth IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[16]  D. Vandermeulen,et al.  Spatially-dense 3D facial asymmetry assessment in both typical and disordered growth , 2011, Journal of anatomy.

[17]  Ghassan Hamarneh,et al.  N-Sift: N-Dimensional Scale Invariant Feature Transform for Matching Medical Images , 2007, ISBI.

[18]  Berk Gökberk,et al.  Nasal Region-Based 3D Face Recognition under Pose and Expression Variations , 2009, ICB.

[19]  T. Theoharis,et al.  Partial matching of interpose 3D facial data for face recognition , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[20]  Mohammed Bennamoun,et al.  Region-based Matching for Robust 3D Face Recognition , 2005, BMVC.

[21]  Liming Chen,et al.  Learning weighted sparse representation of encoded facial normal information for expression-robust 3D face recognition , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[22]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[23]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[24]  Barbara Caputo,et al.  Local velocity-adapted motion events for spatio-temporal recognition , 2007, Comput. Vis. Image Underst..

[25]  Hao Zhang,et al.  Expression-insensitive 3D face recognition using sparse representation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Sridha Sridharan,et al.  Combined 2D/3D Face Recognition Using Log-Gabor Templates , 2006, 2006 IEEE International Conference on Video and Signal Based Surveillance.

[27]  Frank B. ter Haar,et al.  A 3D face matching framework for facial curves , 2009, Graph. Model..

[28]  Patrick J. Flynn,et al.  A Region Ensemble for 3-D Face Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[29]  Alberto Del Bimbo,et al.  Description and retrieval of 3D face models using iso-geodesic stripes , 2006, MIR '06.

[30]  Thomas A. Funkhouser,et al.  The Princeton Shape Benchmark , 2004, Proceedings Shape Modeling Applications, 2004..

[31]  Zhaohui Wu,et al.  Exploring Facial Expression Effects in 3D Face Recognition Using Partial ICP , 2006, ACCV.

[32]  Alberto Del Bimbo,et al.  3D Face Recognition Using Isogeodesic Stripes , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  LoTsz-Wai Rachel,et al.  Local feature extraction and matching on range images , 2009 .

[34]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[35]  S. Malassiotis,et al.  Expression-Compensated 3D Face Recognition with Geodesically Aligned Bilinear Models , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[36]  Mohammed Bennamoun,et al.  An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Paul Suetens,et al.  A Comparative Study of 3-D Face Recognition Under Expression Variations , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[38]  Tieniu Tan,et al.  Automatic 3D face recognition from depth and intensity Gabor features , 2009, Pattern Recognit..

[39]  Luuk J. Spreeuwers,et al.  Fast and Accurate 3D Face Recognition , 2011, International Journal of Computer Vision.

[40]  J. Paul Siebert,et al.  Local feature extraction and matching on range images: 2.5D SIFT , 2009, Comput. Vis. Image Underst..

[42]  Alexander M. Bronstein,et al.  Three-Dimensional Face Recognition , 2005, International Journal of Computer Vision.

[43]  Wei-Yang Lin,et al.  Fusion of Multiple Facial Regions for Expression-Invariant Face Recognition , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[44]  Patrick J. Flynn,et al.  3D Face Recognition with Region Committee Voting , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[45]  Mohammed Bennamoun,et al.  A survey of approaches to three-dimensional face recognition , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[46]  Yunhong Wang,et al.  3D Face Recognition Based on Local Shape Patterns and Sparse Representation Classifier , 2011, MMM.

[47]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[48]  Gérard G. Medioni,et al.  Performance of Geometrix ActiveID^TM 3D Face Recognition Engine on the FRGC Data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[49]  Ioannis A. Kakadiaris,et al.  Using Facial Symmetry to Handle Pose Variations in Real-World 3D Face Recognition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Phalguni Gupta,et al.  An Efficient 3D Face Recognition Algorithm , 2009, 2009 3rd International Conference on New Technologies, Mobility and Security.

[51]  H. Krim,et al.  3D Face Recognition using Euclidean Integral Invariants Signature , 2007, 2007 IEEE/SP 14th Workshop on Statistical Signal Processing.

[52]  Ryutarou Ohbuchi,et al.  SHREC’08 entry: Local volumetric features for 3D model retrieval , 2008, 2008 IEEE International Conference on Shape Modeling and Applications.

[53]  Maurício Pamplona Segundo,et al.  3D Face Recognition Using Simulated Annealing and the Surface Interpenetration Measure , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Frank B. ter Haar,et al.  SHREC 2009 - Shape Retrieval Contest , 2009, 3DOR@Eurographics.

[55]  Ioannis A. Kakadiaris,et al.  Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[57]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[58]  Arman Savran,et al.  3D Face Recognition Benchmarks on the Bosphorus Database with Focus on Facial Expressions , 2008, BIOID.

[59]  Remco C. Veltkamp,et al.  A Survey of 3D Face Recognition Methods , 2005, AVBPA.

[60]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[61]  L. Akarun,et al.  A 3D Face Recognition System for Expression and Occlusion Invariance , 2008, 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems.

[62]  Paul Suetens,et al.  Isometric deformation invariant 3D shape recognition , 2012, Pattern Recognit..

[63]  Paul Suetens,et al.  Isometric deformation modeling using singular value decomposition for 3D expression-invariant face recognition , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[64]  R. Horaud,et al.  Surface feature detection and description with applications to mesh matching , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Faisal R. Al-Osaimi,et al.  An Expression Deformation Approach to Non-rigid 3D Face Recognition , 2009, International Journal of Computer Vision.

[66]  Paul Suetens,et al.  Symmetric surface-feature based 3D face recognition for partial data , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[67]  Yanfeng Sun,et al.  3D face recognition based on sparse representation , 2010, The Journal of Supercomputing.

[68]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[69]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[70]  Ping Yan,et al.  Empirical Evaluation of Advanced Ear Biometrics , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[71]  Alberto Del Bimbo,et al.  3D partial face matching using local shape descriptors , 2011, J-HGBU '11.

[72]  Thomas Vetter,et al.  Expression invariant 3D face recognition with a Morphable Model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.