Deep Learning in Face Recognition Across Variations in Pose and Illumination

Even though face recognition in frontal view and normal lighting conditions works very well, the performance drops sharply in extreme conditions. Recently there is plenty of work dealing with pose and illumination problems, respectively. However both the lighting and pose variations always happen simultaneously in general conditions, and consequently we propose an end-to-end face recognition algorithm to deal with two variations at the same time based on convolutional neural networks. In order to achieve better performance, we extract discriminative nonlinear features that are invariant to pose and illumination. We propose to use the 1 × 1 convolutional kernels to extract the local features. Furthermore a parallel multi-stream convolutional neural network is developed to extract multi-hierarchy features which are more efficient than single-scale features. In the experiments we obtain the average face recognition rate of 96.9% on MultiPIE dataset. Even for profile position, the average recognition rate is also around 98.5% in different lighting conditions, which improves the state-of-the-art face recognition across poses and illumination by 7.5%.

[1]  Sébastien Marcel,et al.  The 2013 speaker recognition evaluation in mobile environment , 2013, 2013 International Conference on Biometrics (ICB).

[2]  Dacheng Tao,et al.  Pose-invariant face recognition with homography-based normalization , 2017, Pattern Recognit..

[3]  Jonghyun Choi,et al.  Robust pose invariant face recognition using coupled latent space discriminant analysis , 2012, Comput. Vis. Image Underst..

[4]  Dimitris N. Metaxas,et al.  Reconstruction-Based Disentanglement for Pose-Invariant Face Recognition , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Jiwen Lu,et al.  Discriminative Deep Metric Learning for Face Verification in the Wild , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Huiyu Zhou,et al.  Combining Perceptual Features With Diffusion Distance for Face Recognition , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[8]  Yongsheng Gao,et al.  Face Recognition Using Line Edge Map , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Amnon Shashua,et al.  The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[11]  Jonathan Warrell,et al.  Tied Factor Analysis for Face Recognition across Large Pose Differences , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  John D. Austin,et al.  Adaptive histogram equalization and its variations , 1987 .

[13]  Gang Hua,et al.  Probabilistic Elastic Matching for Pose Variant Face Verification , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Jian Yang,et al.  KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[16]  Norbert Krüger,et al.  Face Recognition by Elastic Bunch Graph Matching , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Xin Liu,et al.  Morphable Displacement Field Based Image Matching for Face Recognition across Pose , 2012, ECCV.

[18]  Xiaogang Wang,et al.  Deeply learned face representations are sparse, selective, and robust , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Richa Singh,et al.  Face Verification via Class Sparsity Based Supervised Encoding , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[22]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[23]  David J. Kriegman,et al.  Acquiring linear subspaces for face recognition under variable lighting , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Wen Gao,et al.  Coupled Bias–Variance Tradeoff for Cross-Pose Face Recognition , 2012, IEEE Transactions on Image Processing.

[25]  Tsuhan Chen,et al.  Learning patch correspondences for improved viewpoint invariant face recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Xiaogang Wang,et al.  Multi-View Perceptron: a Deep Model for Learning Face Identity and View Representations , 2014, NIPS.

[27]  D. Jacobs,et al.  Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch , 2011, CVPR 2011.

[28]  Ying Li,et al.  Spherical harmonic based linear face de-lighting and compensation , 2007, Appl. Math. Comput..

[29]  Shiguang Shan,et al.  Multi-view Deep Network for Cross-View Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  David J. Kriegman,et al.  Pose, illumination and expression invariant pairwise face-similarity measure via Doppelgänger list comparison , 2011, 2011 International Conference on Computer Vision.

[32]  Dacheng Tao,et al.  Multi-Task Pose-Invariant Face Recognition , 2015, IEEE Transactions on Image Processing.

[33]  Dimitris N. Metaxas,et al.  Reconstruction for Feature Disentanglement in Pose-invariant Face Recognition , 2017, ArXiv.

[34]  Xiaoming Liu,et al.  Disentangled Representation Learning GAN for Pose-Invariant Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  David W. Jacobs,et al.  Generalized Multiview Analysis: A discriminative latent space , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Joel Z. Leibo,et al.  Learning invariant representations and applications to face verification , 2013, NIPS.

[38]  Jonghyun Choi,et al.  Multi-Directional Multi-Level Dual-Cross Patterns for Robust Face Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Shengcai Liao,et al.  Partial Face Recognition: Alignment-Free Approach , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  David J. Kriegman,et al.  What Is the Set of Images of an Object Under All Possible Illumination Conditions? , 1998, International Journal of Computer Vision.

[41]  Ming Shao,et al.  Random Faces Guided Sparse Many-to-One Encoder for Pose-Invariant Face Recognition , 2013, 2013 IEEE International Conference on Computer Vision.

[42]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Shih-Chia Huang,et al.  Efficient Contrast Enhancement Using Adaptive Gamma Correction With Weighting Distribution , 2013, IEEE Transactions on Image Processing.

[44]  Shiguang Shan,et al.  Maximizing intra-individual correlations for face recognition across pose differences , 2009, CVPR.

[45]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Rama Chellappa,et al.  Pose-Invariant Face Recognition Using Markov Random Fields , 2013, IEEE Transactions on Image Processing.

[47]  Ronen Basri,et al.  Lambertian Reflectance and Linear Subspaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[49]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[50]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[51]  Xiaoyue Jiang,et al.  Learning from Real Images to Model Lighting Variations for Face Images , 2008, ECCV.

[52]  Shiguang Shan,et al.  Multi-View Discriminant Analysis , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Xiaogang Wang,et al.  Deep Learning Identity-Preserving Face Space , 2013, 2013 IEEE International Conference on Computer Vision.

[54]  Xiaoyue Jiang,et al.  Lighting Alignment for Image Sequences , 2015, ICIG.

[55]  Gérard G. Medioni,et al.  Pose-Aware Face Recognition in the Wild , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Xiaoyang Tan,et al.  Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions , 2007, IEEE Transactions on Image Processing.

[57]  Jian Sun,et al.  Blessing of Dimensionality: High-Dimensional Feature and Its Efficient Compression for Face Verification , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Bir Bhanu,et al.  Reference Face Graph for Face Recognition , 2014, IEEE Transactions on Information Forensics and Security.

[59]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Hossein Mobahi,et al.  Toward a Practical Face Recognition System: Robust Alignment and Illumination by Sparse Representation , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[62]  Patrick J. Flynn,et al.  Pose-robust recognition of low-resolution face images , 2013, CVPR 2011.

[63]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[64]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Carlos D. Castillo,et al.  Using Stereo Matching with General Epipolar Geometry for 2D Face Recognition across Pose , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  P. Hanrahan,et al.  On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[67]  Jian-Huang Lai,et al.  Normalization of Face Illumination Based on Large-and Small-Scale Features , 2011, IEEE Transactions on Image Processing.

[68]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[69]  Shiguang Shan,et al.  Stacked Progressive Auto-Encoders (SPAE) for Face Recognition Across Poses , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.