Location-sensitive sparse representation of deep normal patterns for expression-robust 3D face recognition

This paper presents a straight-forward yet efficient, and expression-robust 3D face recognition approach by exploring location sensitive sparse representation of deep normal patterns (DNP). In particular, given raw 3D facial surfaces, we first run 3D face pre-processing pipeline, including nose tip detection, face region cropping, and pose normalization. The 3D coordinates of each normalized 3D facial surface are then projected into 2D plane to generate geometry images, from which three images of facial surface normal components are estimated. Each normal image is then fed into a pre-trained deep face net to generate deep representations of facial surface normals, i.e., deep normal patterns. Considering the importance of different facial locations, we propose a location sensitive sparse representation classifier (LS-SRC) for similarity measure among deep normal patterns associated with different 3D faces. Finally, simple score-level fusion of different normal components are used for the final decision. The proposed approach achieves significantly high performance, and reporting rank-one scores of 98.01%, 97.60%, and 96.13% on the FRGC v2.0, Bosphorus, and BU-3DFE databases when only one sample per subject is used in the gallery. These experimental results reveals that the performance of 3D face recognition would be constantly improved with the aid of training deep models from massive 2D face images, which opens the door for future directions of 3D face recognition.

[1]  Ioannis A. Kakadiaris,et al.  Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Yunhong Wang,et al.  Lock3DFace: A large-scale database of low-cost Kinect 3D faces , 2016, 2016 International Conference on Biometrics (ICB).

[3]  Arman Savran,et al.  3D Face Recognition Benchmarks on the Bosphorus Database with Focus on Facial Expressions , 2008, BIOID.

[4]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Paul Suetens,et al.  meshSIFT: Local surface features for 3D face recognition under expression variations and partial data , 2013, Comput. Vis. Image Underst..

[6]  Vinod Chandran,et al.  3D Face Recognition using Log-Gabor Templates , 2006, BMVC.

[7]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[9]  Anil K. Jain,et al.  Deformation Modeling for Robust 3D Face Matching , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Ioannis A. Kakadiaris,et al.  Twins 3D face recognition challenge , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[11]  Di Huang,et al.  3-D Face Recognition Using eLBP-Based Facial Description and Local Feature Hybrid Matching , 2012, IEEE Transactions on Information Forensics and Security.

[12]  Berk Gökberk,et al.  Regional Registration for Expression Resistant 3-D Face Recognition , 2010, IEEE Transactions on Information Forensics and Security.

[13]  Liming Chen,et al.  A group of facial normal descriptors for recognizing 3D identical twins , 2012, 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[14]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[15]  Alberto Del Bimbo,et al.  Sparse Matching of Salient Facial Curves for Recognition of 3-D Faces With Missing Parts , 2013, IEEE Transactions on Information Forensics and Security.

[16]  Liming Chen,et al.  Towards 3D Face Recognition in the Real: A Registration-Free Approach Using Fine-Grained Matching of 3D Keypoint Descriptors , 2014, International Journal of Computer Vision.

[17]  Anuj Srivastava,et al.  Three-Dimensional Face Recognition Using Shapes of Facial Curves , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Xiangyu Zhu,et al.  Face Alignment in Full Pose Range: A 3D Total Solution , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  L. Spreeuwers.  Fast and Accurate 3 D Face Recognition Using Registration to an Intrinsic Coordinate System and Fusion of Multiple Region Classifiers , 2011 .

[20]  Michael G. Strintzis,et al.  Bilinear Models for 3-D Face and Facial Expression Recognition , 2008, IEEE Transactions on Information Forensics and Security.

[21]  Timo Bolkart,et al.  A Robust Multilinear Model Learning Framework for 3D Faces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Alexander M. Bronstein,et al.  Expression-Invariant Representations of Faces , 2007, IEEE Transactions on Image Processing.

[23]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Dimitrios Hatzinakos,et al.  Iterative Closest Normal Point for 3D Face Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Jonghyun Choi,et al.  Multi-Directional Multi-Level Dual-Cross Patterns for Robust Face Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Mohammed Bennamoun,et al.  An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Patrick J. Flynn,et al.  A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition , 2006, Comput. Vis. Image Underst..

[29]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[30]  Paul Suetens,et al.  A Comparative Study of 3-D Face Recognition Under Expression Variations , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[31]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[32]  Volker Blanz,et al.  Automated 3D Face Reconstruction from Multiple Images Using Quality Measures , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Maurício Pamplona Segundo,et al.  3D Face Recognition Using Simulated Annealing and the Surface Interpenetration Measure , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Ioannis A. Kakadiaris,et al.  UR3D-C: Linear dimensionality reduction for efficient 3D face recognition , 2011, 2011 International Joint Conference on Biometrics (IJCB).

[35]  Thomas Vetter,et al.  Expression invariant 3D face recognition with a Morphable Model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[36]  Anil K. Jain,et al.  Segmentation and Classification of Range Images , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Hassen Drira,et al.  3D Face Recognition under Expressions, Occlusions, and Pose Variations , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Alberto Del Bimbo,et al.  3D Face Recognition Using Isogeodesic Stripes , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Jie Chen,et al.  Fusing Local Patterns of Gabor Magnitude and Phase for Face Recognition , 2010, IEEE Transactions on Image Processing.

[41]  Patrick J. Flynn,et al.  Multiple Nose Region Matching for 3D Face Recognition under Varying Facial Expression , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Shengcai Liao,et al.  Learning Face Representation from Scratch , 2014, ArXiv.

[43]  Jian Sun,et al.  Bayesian Face Revisited: A Joint Formulation , 2012, ECCV.

[44]  Hao Zhang,et al.  Expression-insensitive 3D face recognition using sparse representation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Huibin Li,et al.  Expression-robust 3D face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns , 2014, Neurocomputing.

[46]  Luuk J. Spreeuwers,et al.  Fast and Accurate 3D Face Recognition , 2011, International Journal of Computer Vision.

[47]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[48]  Jean-Luc Dugelay,et al.  An Efficient LBP-Based Descriptor for Facial Depth Images Applied to Gender Recognition Using RGB-D Face Data , 2012, ACCV Workshops.

[49]  Yan Wang,et al.  EI3D: Expression-invariant 3D face recognition based on feature and shape matching , 2016, Pattern Recognit. Lett..

[50]  Xiaoou Tang,et al.  Robust 3D Face Recognition by Local Shape Difference Boosting , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Abolghasem A. Raie,et al.  2.5D face recognition using Patch Geodesic Moments , 2012, Pattern Recognit..

[52]  Mohammed Bennamoun,et al.  Keypoint Detection and Local Feature Matching for Textured 3D Face Recognition , 2007, International Journal of Computer Vision.

[53]  Alberto Del Bimbo,et al.  Superfaces: A Super-Resolution Model for 3D Faces , 2012, ECCV Workshops.

[54]  Anuj Srivastava,et al.  An Intrinsic Framework for Analysis of Facial Surfaces , 2009, International Journal of Computer Vision.

[55]  Mehryar Emambakhsh,et al.  Nasal Patches and Curves for Expression-Robust 3D Face Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Faisal R. Al-Osaimi,et al.  An Expression Deformation Approach to Non-rigid 3D Face Recognition , 2009, International Journal of Computer Vision.

[57]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.