Achieving robust face recognition from video by combining a weak photometric model and a learnt generic face invariant

In spite of over two decades of intense research, illumination and pose invariance remain prohibitively challenging aspects of face recognition for most practical applications. The objective of this work is to recognize faces using video sequences both for training and recognition input, in a realistic, unconstrained setup in which lighting, pose and user motion pattern have a wide variability and face images are of low resolution. The central contribution is an illumination invariant, which we show to be suitable for recognition from video of loosely constrained head motion. In particular there are three contributions: (i) we show how a photometric model of image formation can be combined with a statistical model of generic face appearance variation to exploit the proposed invariant and generalize in the presence of extreme illumination changes; (ii) we introduce a video sequence ''re-illumination'' algorithm to achieve fine alignment of two video sequences; and (iii) we use the smoothness of geodesically local appearance manifold structure and a robust same-identity likelihood to achieve robustness to unseen head poses. We describe a fully automatic recognition system based on the proposed method and an extensive evaluation on 323 individuals and 1474 video sequences with extreme illumination, pose and head motion variation. Our system consistently achieved a nearly perfect recognition rate (over 99.7% on all four databases).

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  Trevor Darrell,et al.  Face Recognition from Long-Term Observations , 2002, ECCV.

[3]  Roberto Cipolla,et al.  A pose-wise linear illumination manifold model for face recognition using video , 2009, Comput. Vis. Image Underst..

[4]  Alex Pentland,et al.  Human Face Recognition and the Face Image Set's Topology , 1994 .

[5]  Roberto Cipolla,et al.  An information-theoretic approach to face recognition from face motion manifolds , 2006, Image Vis. Comput..

[6]  Roberto Cipolla,et al.  A methodology for rapid illumination-invariant face recognition using image processing filters , 2009, Comput. Vis. Image Underst..

[7]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[8]  Masashi Nishiyama,et al.  Face recognition using the classified appearance-based quotient image , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[9]  Trevor Darrell,et al.  Face recognition with image sets using manifold density divergence , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[10]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[11]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[12]  Amnon Shashua,et al.  The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Roberto Cipolla,et al.  Automatic Cast Listing in Feature-Length Films with Anisotropic Manifold Space , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  Hiroshi Murase,et al.  Visual learning and recognition of 3-d objects from appearance , 2005, International Journal of Computer Vision.

[16]  Roberto Cipolla,et al.  Incremental Learning of Temporally-Coherent Gaussian Mixture Models , 2005, BMVC.

[17]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[18]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[19]  Tsuhan Chen,et al.  Video-based face recognition using adaptive hidden Markov models , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[20]  Dariu Gavrila,et al.  Pedestrian Detection from a Moving Vehicle , 2000, ECCV.

[21]  Guo Jun.  Solution of small sample size problem in face recognition using Gabor wavelet transform , 2007 .

[22]  Ken-ichi Maeda,et al.  Towards 3-Dimensional Pattern Recognition , 2004, SSPR/SPR.

[23]  Ralph Gross,et al.  Generic vs. person specific active appearance models , 2005, Image Vis. Comput..

[25]  Ognjen Arandjelovic.  Computationally efficient application of the generic shape-illumination invariant to face recognition from video , 2012, Pattern Recognit..

[26]  Björn Stenger,et al.  Filtering using a tree-based estimator , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[27]  Bayya Yegnanarayana,et al.  Real time face recognition system using autoassociative neural network models , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[28]  Tae-Kyun Kim,et al.  Boosted manifold principal angles for image set-based recognition , 2007, Pattern Recognit..

[29]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Lei Zhang,et al.  Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Alice J. O'Toole,et al.  Face Recognition Vendor Test 2006 and Iris Challenge Evaluation 2006 Large-Scale Results | NIST , 2007 .

[32]  Ognjen Arandjelovic,et al.  Recognition from Appearance Subspaces across Image Sets of Variable Scale , 2010, BMVC.

[33]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Rama Chellappa,et al.  Probabilistic recognition of human faces from video , 2002, Proceedings. International Conference on Image Processing.

[35]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[36]  Ralph R. Martin,et al.  Merging and Splitting Eigenspace Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  AkamatsuShigeru,et al.  How Should We RepresentFaces for Automatic Recognition , 1999 .

[38]  Andrew Zisserman,et al.  Person Spotting: Video Shot Retrieval for Face Sets , 2005, CIVR.

[39]  Andrew W. Fitzgibbon,et al.  On Affine Invariant Clustering and Automatic Cast Listing in Movies , 2002, ECCV.

[40]  Haitao Wang,et al.  Face recognition under varying lighting conditions using self quotient image , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[41]  Lior Wolf,et al.  Learning over Sets using Kernel Principal Angles , 2003, J. Mach. Learn. Res..

[42]  Mark S. Nixon,et al.  Extending the Feature Vector for Automatic Face Recognition , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Dong Xu,et al.  Multilinear Discriminant Analysis for Face Recognition , 2007, IEEE Transactions on Image Processing.

[44]  Alexander H. Waibel,et al.  Growing Gaussian mixture models for pose invariant face recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[45]  Xiaogang Wang,et al.  Unified subspace analysis for face recognition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[46]  Mongi A. Abidi,et al.  Face recognition: evaluation report for FaceIt identification and surveillance , 2003, International Conference on Quality Control by Artificial Vision.

[47]  Terence Sim,et al.  Exploring Face Space , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[48]  Xiaogang Wang,et al.  Bayesian face recognition based on Gaussian mixture models , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[49]  David J. Kriegman,et al.  Illumination cones for recognition under variable lighting: faces , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[50]  J. Melo,et al.  Overview and summary , 1985 .

[51]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[52]  Andrew Zisserman,et al.  Automatic face recognition for film character retrieval in feature-length films , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[53]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[54]  Osamu Yamaguchi,et al.  Face Recognition Using Multi-viewpoint Patterns for Robot Vision , 2003, ISRR.

[55]  Lei Zhang,et al.  Face synthesis and recognition from a single image under arbitrary unknown lighting using a spherical harmonic basis morphable model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[56]  Jorma Rissanen,et al.  Minimum Description Length Principle , 2010, Encyclopedia of Machine Learning.

[57]  Michael Elad,et al.  A Variational Framework for Retinex , 2002, IS&T/SPIE Electronic Imaging.

[58]  Bayya Yegnanarayana,et al.  Real time face authentication system using autoassociative neural network models , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[59]  Wen Gao,et al.  Histogram of Gabor Phase Patterns (HGPP): A Novel Object Representation Approach for Face Recognition , 2007, IEEE Transactions on Image Processing.

[60]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[61]  Yee Whye Teh,et al.  Names and faces in the news , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[62]  David J. Kriegman,et al.  Online learning of probabilistic appearance manifolds for video-based recognition and tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[63]  Roberto Cipolla,et al.  An Illumination Invariant Face Recognition System for Access Control using Video , 2004, BMVC.

[64]  Martin D. Levine,et al.  State-of-the-art of 3D facial reconstruction methods for face recognition based on a single 2D training image per person , 2009, Pattern Recognit. Lett..

[65]  D.O. Gorodnichy,et al.  Associative neural networks as means for low-resolution video-based recognition , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[66]  David J. Kriegman,et al.  What Is the Set of Images of an Object Under All Possible Illumination Conditions? , 1998, International Journal of Computer Vision.