A bi-modal face recognition framework integrating facial expression with facial appearance

Among many biometric characteristics, the facial biometric is considered to be the least intrusive technology that can be deployed in the real-world visual surveillance environment. However, in facial biometric, little research attention has been paid to facial expression changes. In fact, facial expression changes have often been treated as noise that would degrade the recognition performance. This paper studies an innovative viewpoint: (1) whether facial expression changes, namely facial behavior, can be positively used for face recognition or not? (2) furthermore, can facial behavior be integrated with facial appearance for assisting the extra-personal separation to enhance face recognition performance? We propose a bi-modal face recognition framework which integrates facial expression with facial appearance. Substantial experiments on multiple facial appearance and facial expression data have been conducted. Our experimental results have validated that facial behavior can play a positive role in face recognition and can assist facial appearance in extra-personal separation in multiple modalities for personal identification improvement.

[1]  Patrick J. Flynn,et al.  An evaluation of multimodal 2D+3D face biometrics , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[3]  Konstantinos N. Plataniotis,et al.  Regularized discriminant analysis for the small sample size problem in face recognition , 2003, Pattern Recognit. Lett..

[4]  Alexander M. Bronstein,et al.  Expression-Invariant Representations of Faces , 2007, IEEE Transactions on Image Processing.

[5]  Yanxi Liu,et al.  Facial asymmetry quantification for expression invariant human identification , 2003, Comput. Vis. Image Underst..

[6]  Lawrence Sirovich,et al.  Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  David G. Stork,et al.  Pattern Classification , 1973 .

[8]  Maja Pantic,et al.  Fully Automatic Facial Action Unit Detection and Temporal Analysis , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[9]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[10]  Venu Govindaraju,et al.  Introduction to the Special Issue on Recent Advances in Biometric Systems [Guest Editorial] , 2007, IEEE Trans. Syst. Man Cybern. Part B.

[11]  Danilo P. Mandic,et al.  Artificial Intelligence for Human Computing , 2007 .

[12]  Sudeep Sarkar,et al.  Comparison and Combination of Ear and Face Images in Appearance-Based Biometrics , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Ja-Chen Lin,et al.  A new LDA-based face recognition system which can solve the small sample size problem , 1998, Pattern Recognit..

[14]  Aleix M. Martínez,et al.  Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Tom Hintz,et al.  Facial behavior as behavior biometric? an empirical study , 2007, 2007 IEEE International Conference on Systems, Man and Cybernetics.

[17]  Trevor Darrell,et al.  Integrated face and gait recognition from multiple views , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[18]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  V. Ivancevic,et al.  Factor analysis of essential facial features , 2003, Proceedings of the 25th International Conference on Information Technology Interfaces, 2003. ITI 2003..

[20]  Jaihie Kim,et al.  Fusion of visual and infra-red face scores by weighted power series , 2008, Pattern Recognit. Lett..

[21]  Xiaoli Zhou,et al.  Integrating Face and Gait for Human Recognition at a Distance in Video , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  Harry Wechsler,et al.  Reliable face recognition methods - system design, implementation and evaluation , 2006 .

[23]  Tom Hintz,et al.  Adaptive Multiple Experts System for personal identification using facial behaviour biometrics , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[24]  Michael J. Lyons,et al.  Automatic Classification of Single Facial Images , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Hyeonjoon Moon,et al.  The FERET Evaluation Methodology for Face-Recognition Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Tony Jan,et al.  Expression-invariant face recognition system using subspace model analysis , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[27]  Ming-Hsuan Yang,et al.  Kernel Eigenfaces vs. Kernel Fisherfaces: Face recognition using kernel methods , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[28]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[29]  H. Bülthoff,et al.  The use of facial motion and facial form during the processing of identity , 2003, Vision Research.

[30]  Arun Ross,et al.  Multibiometric systems , 2004, CACM.

[31]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[32]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[33]  Tom Hintz,et al.  An evaluation of bi-modal facial appearance+facial expression face biometrics , 2008, 2008 19th International Conference on Pattern Recognition.

[34]  Konstantinos N. Plataniotis,et al.  Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition , 2005, Pattern Recognit. Lett..

[35]  Alex Pentland,et al.  Human Computing and Machine Understanding of Human Behavior: A Survey , 2007, Artifical Intelligence for Human Computing.