Integrating Illumination, Motion, and Shape Models for Robust Face Recognition in Video

The use of video sequences for face recognition has been relatively less studied compared to image-based approaches. In this paper, we present an analysis-by-synthesis framework for face recognition from video sequences that is robust to large changes in facial pose and lighting conditions. This requires tracking the video sequence, as well as recognition algorithms that are able to integrate information over the entire video; we address both these problems. Our method is based on a recently obtained theoretical result that can integrate the effects of motion, lighting, and shape in generating an image using a perspective camera. This result can be used to estimate the pose and structure of the face and the illumination conditions for each frame in a video sequence in the presence of multiple point and extended light sources. We propose a new inverse compositional estimation approach for this purpose. We then synthesize images using the face model estimated from the training data corresponding to the conditions in the probe sequences. Similarity between the synthesized and the probe images is computed using suitable distance measurements. The method can handle situations where the pose and lighting conditions in the training and testing data are completely disjoint. We show detailed performance analysis results and recognition scores on a large video dataset.

[1]  P. Hanrahan,et al.  On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  Amit K. Roy-Chowdhury,et al.  Inverse Compositional Estimation of 3D Pose And Lighting in Dynamic Scenes , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Stefano Soatto,et al.  Real-Time Feature Tracking and Outlier Rejection with Changes in Illumination , 2001, ICCV.

[4]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[6]  Rama Chellappa,et al.  Face Processing: Advanced Modeling and Methods , 2006, J. Electronic Imaging.

[7]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression Database , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Jing Xiao,et al.  Multi-view AAM fitting and camera calibration , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[9]  Roberto Cipolla,et al.  An Illumination Invariant Face Recognition System for Access Control using Video , 2004, BMVC.

[10]  Pradeep K. Khosla,et al.  "Corefaces" - robust shift invariant PCA based correlation filter for illumination tolerant face recognition , 2004, CVPR 2004.

[11]  Alice J. O'Toole,et al.  A video database of moving faces and people , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Lei Zhang,et al.  Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Gregory D. Hager,et al.  Efficient Region Tracking With Parametric Models of Geometry and Illumination , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Witold Pedrycz,et al.  Face recognition: A study in information fusion using fuzzy integral , 2005, Pattern Recognit. Lett..

[15]  Peter Eisert,et al.  Illumination Compensated Motion Estimation for Analysis Synthesis Coding , 1996 .

[16]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[17]  Ralph Gross,et al.  Appearance-based face recognition and light-fields , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[19]  Trevor Darrell,et al.  Face recognition with image sets using manifold density divergence , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[21]  P J. Phillips,et al.  Face Recognition Vendor Test 2000: Evaluation Report , 2001 .

[22]  A. Roy-Chowdhury,et al.  Pose and Illumination Invariant Registration and Tracking for Video-based Face Recognition , 2006 .

[23]  Avinash C. Kak,et al.  Robust motion estimation under varying illumination , 2005, Image Vis. Comput..

[24]  Amit K. Roy-Chowdhury,et al.  Integrating the effects of motion, illumination and structure in video sequences , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[25]  Rama Chellappa,et al.  Face reconstruction from monocular video using uncertainty analysis and a generic model , 2003, Comput. Vis. Image Underst..

[26]  B. V. K. Vijaya Kumar,et al.  "Corefaces" - robust shift invariant PCA based correlation filter for illumination tolerant face recognition , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[27]  Tsuhan Chen,et al.  Video-based face recognition using adaptive hidden Markov models , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[28]  Tsuhan Chen,et al.  Learning Patch Dependencies for Improved Pose Mismatched Face Verification , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[29]  Amit K. Roy-Chowdhury,et al.  Integrating Motion, Illumination, and Structure in Video Sequences with Applications in Illumination-Invariant Tracking , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Ravi Ramamoorthi,et al.  Modeling Illumination Variation with Spherical Harmonics , 2005 .

[31]  Jeffrey Ho,et al.  On the Effect of Illumination and Face Recognition , 2005 .

[32]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[33]  P. Jonathon Phillips,et al.  Face recognition based on frontal views generated from non-frontal images , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[34]  B. V. K. Vijaya Kumar,et al.  A Still-to-Video Face Verification System Using Advanced Correlation Filters , 2004, ICBA.

[35]  James H. Elder,et al.  Probabilistic Linear Discriminant Analysis for Inferences About Identity , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[36]  Patrick J. Flynn,et al.  Overview of the face recognition grand challenge , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[37]  Trevor Darrell,et al.  Face Recognition from Long-Term Observations , 2002, ECCV.

[38]  Patrick J. Flynn,et al.  A Survey Of 3D and Multi-Modal 3D+2D Face Recognition , 2004 .

[39]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[40]  Andrew Zisserman,et al.  Identifying individuals in video by combining 'generative' and discriminative head models , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[41]  Demetri Terzopoulos,et al.  Multilinear independent components analysis , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).