Face recognition: face in video, age invariance, and facial marks

Automatic face recognition has been extensively studied over the past decades in various domains (e.g., 2D, 3D, and video) resulting in a dramatic improvement. However, face recognition performance severely degrades under pose, lighting and expression variations, occlusion, and aging. Pose and lighting variations along with low image resolutions are major sources of degradation of face recognition performance in surveillance video. We propose a video-based face recognition framework using 3D face modeling and Pan-Tilt-Zoom (PTZ) cameras to overcome the pose/lighting variations and low resolution problems. We propose a 3D aging modeling technique and show how it can be used to compensate for age variations to improve face recognition performance. The aging modeling technique adapts view invariant 3D face models to the given 2D face aging database. We also propose an automatic facial mark detection method and a fusion scheme that combines the facial mark matching with a commercial face recognition matcher. The proposed approach can be used (i) as an indexing scheme for a face image retrieval system and (ii) to augment global facial features to improve the recognition performance. Experimental results show (i) high recognition accuracy (>99%) on a large scale video data (>200 subjects), (ii) ∼10% improvement in recognition accuracy using the proposed aging model, and (iii) ∼0.94% improvement in the recognition accuracy by utilizing facial marks.

[1]  Thomas Serre,et al.  Component-based face detection , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[2]  L. Grafakos.  Classical and modern Fourier analysis , 2003 .

[3]  Arun Ross,et al.  Score normalization in multimodal biometric systems , 2005, Pattern Recognit..

[4]  James M. Rehg,et al.  Fast Asymmetric Learning for Cascade Face Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Edward Y. Chang,et al.  Invariant feature extraction and biased statistical inference for video surveillance , 2003, Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, 2003..

[6]  Wen Gao,et al.  Local Linear Regression (LLR) for Pose Invariant Face Recognition , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[7]  B. K. Julsing,et al.  Face Recognition with Local Binary Patterns , 2012 .

[8]  Josef Kittler,et al.  Component-based LDA face description for image retrieval and MPEG-7 standardisation , 2005, Image Vis. Comput..

[9]  Jing Xiao,et al.  Real-time combined 2D+3D active appearance models , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[10]  Timothy F. Cootes,et al.  View-based active appearance models , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[11]  Hong Yan,et al.  An Analytic-to-Holistic Approach for Face Recognition Based on a Single Frontal View , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  David J. Kriegman,et al.  Face Recognition Using 3-D Models: Pose and Illumination , 2006, Proceedings of the IEEE.

[13]  Yair Weiss,et al.  Learning object detection from a small number of examples: the importance of good features , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[14]  Thomas Vetter,et al.  Skin Detail Analysis for Face Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[16]  Rama Chellappa,et al.  Face reconstruction from monocular video using uncertainty analysis and a generic model , 2003, Comput. Vis. Image Underst..

[17]  Songcan Chen,et al.  Adaptively weighted sub-pattern PCA for face recognition , 2005, Neurocomputing.

[18]  Rama Chellappa,et al.  A system identification approach for video-based face recognition , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[19]  Song-Chun Zhu,et al.  A Multi-Resolution Dynamic Model for Face Aging Simulation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  David Salesin,et al.  Modeling and Animating Realistic Faces from Images , 2002, International Journal of Computer Vision.

[21]  Yajie Tian,et al.  Handbook of face recognition , 2003 .

[22]  Matti Pietikäinen,et al.  Face Recognition with Local Binary Patterns , 2004, ECCV.

[23]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[24]  Jing Xiao,et al.  A Closed-Form Solution to Non-Rigid Shape and Motion Recovery , 2004, International Journal of Computer Vision.

[25]  Arun Ross,et al.  Face Recognition in Video: Adaptive Fusion of Multiple Matchers , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Linda G. Shapiro,et al.  Computer Vision , 2001 .

[27]  Ralph Gross,et al.  Active appearance models with occlusion , 2006, Image Vis. Comput..

[28]  David J. Kriegman,et al.  From few to many: generative models for recognition under variable pose and illumination , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[29]  Rama Chellappa,et al.  Probabilistic recognition of human faces from video , 2002, Proceedings. International Conference on Image Processing.

[30]  R. Fisher.  THE STATISTICAL UTILIZATION OF MULTIPLE MEASUREMENTS , 1938 .

[31]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Mubarak Shah,et al.  KNIGHT/spl trade/: a real time surveillance system for multiple and non-overlapping cameras , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[33]  Rama Chellappa,et al.  Principal components null space analysis for image and video classification , 2006, IEEE Transactions on Image Processing.

[34]  Tomaso A. Poggio,et al.  Face recognition: component-based versus global approaches , 2003, Comput. Vis. Image Underst..

[35]  M. Shah,et al.  KNIGHT M : A REAL TIME SURVEILLANCE SYSTEM FOR MULTIPLE OVERLAPPING AND NON-OVERLAPPING CAMERAS , 2003 .

[36]  Tsuhan Chen,et al.  The CMU Face In Action (FIA) Database , 2005, AMFG.

[37]  P. Jonathon Phillips,et al.  Face recognition vendor test 2002 , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[38]  P J. Phillips,et al.  Face Recognition Vendor Test 2000: Evaluation Report , 2001 .

[39]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[40]  K. Ricanek,et al.  Aspects of Age Variation in Facial Morphology Affecting Biometrics , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[41]  Michael J. Brooks,et al.  Issues in Automated Visual Surveillance , 2003, DICTA.

[42]  Yiying Tong,et al.  Face recognition with temporal invariance: A 3D aging model , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[43]  Ralph Gross,et al.  Generic vs. person specific active appearance models , 2005, Image Vis. Comput..

[44]  Rama Chellappa,et al.  Face Verification Across Age Progression , 2006, IEEE Transactions on Image Processing.

[45]  Norbert Krüger,et al.  Face recognition by elastic bunch graph matching , 1997, Proceedings of International Conference on Image Processing.

[46]  Frederick I. Parke,et al.  Computer generated animation of faces , 1972, ACM Annual Conference.

[47]  J. B. Pittenger,et al.  Aging faces as viscal-elastic events: implications for a theory of nonrigid shape perception. , 1975, Journal of experimental psychology. Human perception and performance.

[48]  Wen Gao,et al.  Face Recognition under Varying Lighting Based on the Probabilistic Model of Gabor Phase , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[49]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[50]  David Beymer,et al.  Face recognition from one example view , 1995, Proceedings of IEEE International Conference on Computer Vision.

[51]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Anil K. Jain,et al.  Soft Biometric Traits for Personal Recognition Systems , 2004, ICBA.

[53]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[55]  Penio S. Penev,et al.  Local feature analysis: A general statistical theory for object representation , 1996 .

[56]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[57]  Hyeonjoon Moon,et al.  The FERET evaluation methodology for face-recognition algorithms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Anil K. Jain,et al.  3D Model-Based Face Recognition in Video , 2007, ICB.

[59]  Anil K. Jain,et al.  Likelihood Ratio-Based Biometric Score Fusion , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  J. Sklansky,et al.  Segmentation of people in motion , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[61]  Jianqin Zhou,et al.  On discrete cosine transform , 2011, ArXiv.

[62]  Dorin Comaniciu,et al.  Total variation models for variable lighting face recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Stan Z. Li,et al.  Tracking and Recognition of Multiple Faces at Distances , 2007, ICB.

[64]  C. Chibelushi,et al.  Facial Expression Recognition : A Brief Tutorial Overview , 2022 .

[65]  M. A. Grudin,et al.  On internal representations in face recognition systems , 2000, Pattern Recognit..

[66]  Nicole A. Spaun.  Forensic Biometrics from Images and Video at the Federal Bureau of Investigation , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[67]  Karl Ricanek,et al.  MORPH: a longitudinal image database of normal adult age-progression , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[68]  Shmuel Peleg,et al.  Image sequence enhancement using sub-pixel displacements , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[69]  Hans-Peter Seidel,et al.  Prediction of Individual Non‐Linear Aging Trajectories of Faces , 2007, Comput. Graph. Forum.

[70]  Ioannis Pavlidis,et al.  Urban surveillance systems: from the laboratory to the commercial world , 2001, Proc. IEEE.

[71]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[72]  A. O'Toole,et al.  Three-Dimensional Caricatures of Human Heads: Distinctiveness and the Perception of Facial Age , 1997, Perception.

[73]  Kazuhiro Hotta.  Robust face recognition under partial occlusion based on support vector machine with local Gaussian summation kernel , 2008, Image Vis. Comput..

[74]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[75]  P. J. Burt,et al.  Change Detection and Tracking Using Pyramid Transform Techniques , 1985, Other Conferences.

[76]  Rama Chellappa,et al.  A feature based approach to face recognition , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[77]  Anil K. Jain,et al.  Face detection in color images , 2002, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[78]  Wen Gao,et al.  Local Gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and recognition , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[79]  Matthew Brand,et al.  A direct method for 3D factorization of nonrigid motion observed in 2D , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[80]  Rama Chellappa,et al.  Human and machine recognition of faces: a survey , 1995, Proc. IEEE.

[81]  Hans-Hellmut Nagel,et al.  Image Sequences - Ten (Octal) Years - from phenomenology towards a Theoretical Foundation , 1988, Int. J. Pattern Recognit. Artif. Intell..

[82]  C. Christodoulou,et al.  Comparing different classifiers for automatic age estimation , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[83]  Tony Lindeberg,et al.  Feature Detection with Automatic Scale Selection , 1998, International Journal of Computer Vision.

[84]  Rama Chellappa,et al.  Illumination-insensitive face recognition using symmetric shape-from-shading , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[85]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[86]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[87]  Alex Pentland,et al.  View-based and modular eigenspaces for face recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[88]  Paola Campadelli,et al.  A Face Recognition System Based on Local Feature Analysis , 2003, AVBPA.

[89]  Frederick I. Parke,et al.  Computer gernerated animation of faces , 1998 .

[90]  Azriel Rosenfeld,et al.  Computer Vision , 1988, Adv. Comput..

[91]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[92]  Vijayan K. Asari,et al.  An improved face recognition technique based on modular PCA approach , 2004, Pattern Recognit. Lett..

[93]  K. Ricanek,et al.  AUTOMATIC REPRESENTATION OF ADULT AGING IN FACIAL IMAGES , 2022 .

[94]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[95]  Thomas S. Huang,et al.  Accurate Head Pose Tracking in Low Resolution Video , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[96]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[97]  Xinggang Lin,et al.  Age simulation for face recognition , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[98]  S. Ullman,et al.  The interpretation of visual motion , 1977 .

[99]  Zhi-Hua Zhou,et al.  Automatic Age Estimation Based on Facial Aging Patterns , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[100]  Timothy F. Cootes,et al.  Toward Automatic Simulation of Aging Effects on Face Images , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[101]  Thomas S. Huang,et al.  Human face detection in a scene , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[102]  Irfan Essa,et al.  A System for Tracking and Recognizing Multiple People with Multiple Cameras , 1998 .

[103]  Gérard G. Medioni,et al.  Performance of Geometrix ActiveID^TM 3D Face Recognition Engine on the FRGC Data , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[104]  Rita Cucchiara,et al.  Detecting Moving Objects, Ghosts, and Shadows in Video Streams , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[105]  Stefano Soatto,et al.  A Study of Face Recognition as People Age , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[106]  C. Cacou.  Anthropometry of the head and face , 1995 .

[107]  J. L. Roux.  An Introduction to the Kalman Filter , 2003 .

[108]  Anil K. Jain,et al.  Deformation Modeling for Robust 3D Face Matching , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[109]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[110]  Tsuhan Chen,et al.  Video-based face recognition using adaptive hidden Markov models , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[111]  Surendra Ranganath,et al.  Pose-invariant face recognition using a 3D deformable model , 2003, Pattern Recognit..

[112]  Ian Craw,et al.  Finding Face Features , 1992, ECCV.

[113]  Rama Chellappa,et al.  Modeling Age Progression in Young Faces , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[114]  G. Porter,et al.  An anatomical and photographic technique for forensic facial identification. , 2000, Forensic science international.

[115]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[116]  Narendra Ahuja,et al.  Detecting Faces in Images: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[117]  Aleix M. Martínez,et al.  Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[118]  D'arcy W. Thompson,et al.  On Growth and Form , 1917, Nature.

[119]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[120]  Anil K. Jain,et al.  Matching 2.5D face scans to 3D models , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[121]  Dahua Lin,et al.  Recognize High Resolution Faces: From Macrocosm to Microcosm , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[122]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.

[123]  Zhi-Hua Zhou,et al.  Image Region Selection and Ensemble for Face Recognition , 2006, Journal of Computer Science and Technology.

[124]  Hans-Hellmut Nagel,et al.  New likelihood test methods for change detection in image sequences , 1984, Comput. Vis. Graph. Image Process..

[125]  Ioannis A. Kakadiaris,et al.  Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[126]  Mikkel B. Stegmann,et al.  The AAM-API: An Open Source Active Appearance Model Implementation , 2003, MICCAI.