Face and expression recognition based on bag of words method considering holistic and local image features

This paper proposes a new framework for extracting facial features based on the bag of words method, and applies it to face and facial expression recognition. Recently, the bag of words method has been successfully used in object recognition. However, for recognition problems of facial images, the orderless collection of local patches in bag of words method cannot provide strongly distinctive information since the object category (face image) is the same. In our work, a new framework based on bag of words is presented to extract discriminative local facial features while maintaining holistic spatial information at the same time. The new method is applied to both face and facial expression recognition. Experimental results show that only using one neutral expression frame per person for training, our method can obtain the best face recognition performance ever on face images of AR database with extreme expressions, variant illuminations, and partial occlusions. For facial expression recognition, the average rate on the Cohn-Kanade database is 96.33%, which also outperforms the state of the arts.

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  Gregory Shakhnarovich,et al.  Face Recognition in Subspaces , 2011, Handbook of Face Recognition.

[3]  Aggelos K. Katsaggelos,et al.  Automatic facial expression recognition using facial animation parameters and multistream HMMs , 2006, IEEE Transactions on Information Forensics and Security.

[4]  Jie Lin,et al.  Robust face recognition using posterior union model based neural networks , 2009 .

[5]  Masahide Kaneko,et al.  Robust Face Recognition Using Block-Based Bag of Words , 2010, 2010 20th International Conference on Pattern Recognition.

[6]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[7]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  Dahua Lin,et al.  Nonparametric Discriminant Analysis for Face Recognition , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Karim Faez,et al.  Face recognition using adaptively weighted patch PZM array from a single exemplar image per person , 2008, Pattern Recognit..

[10]  Hong Yan,et al.  Face recognition using the weighted fractal neighbor distance , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Jiri Matas,et al.  XM2VTSDB: The Extended M2VTS Database , 1999 .

[12]  Stephen Lin,et al.  Rank-one Projections with Adaptive Margins for Face Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[13]  Ioannis Pitas,et al.  Texture and shape information fusion for facial expression and facial action unit recognition , 2008, Pattern Recognit..

[14]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[15]  Zhi-Hua Zhou,et al.  Recognizing partially occluded, expression variant faces from single training image per person with SOM and soft k-NN ensemble , 2005, IEEE Transactions on Neural Networks.

[16]  Stephen Lin,et al.  Rank-One Projections With Adaptive Margins for Face Recognition , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Kin-Man Lam,et al.  Gabor-based kernel PCA with doubly nonlinear mapping for face recognition with a single face image , 2006, IEEE Transactions on Image Processing.

[20]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  B. K. Julsing,et al.  Face Recognition with Local Binary Patterns , 2012 .

[22]  A. Martínez,et al.  The AR face databasae , 1998 .

[23]  Masahide Kaneko,et al.  Facial Expression Recognition Using Facial-component-based Bag of Words and PHOG Descriptors , 2010 .

[24]  Norbert Krüger,et al.  Face recognition by elastic bunch graph matching , 1997, Proceedings of International Conference on Image Processing.

[25]  Aleix M. Martínez,et al.  Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[27]  Masahide Kaneko,et al.  顔部品の「Bag of Words」とPHOG記述子を用いた顔表情認識 , 2010 .

[28]  Andrea Lagorio,et al.  On the Use of SIFT Features for Face Authentication , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[29]  Gwen Littlewort,et al.  Dynamics of Facial Expression Extracted Automatically from Video , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[30]  Qiang Ji,et al.  A Comparative Study of Local Matching Approach for Face Recognition , 2007, IEEE Transactions on Image Processing.