Joint regularized nearest points for image set based face recognition

Face recognition based on image set has attracted much attention due to its promising performance to overcome various variations. Recently, classifiers of regularized nearest points, including sparse approximated nearest points (SANP), regularized nearest points (RNP) and collaborative regularized nearest points (CRNP), have achieved state-of-the-art performance for image set based face recognition. From a query set and a single-class gallery set, SANP and RNP both generate a pair of nearest points, between which the distance is regarded as the between-set distance. However, the computing of nearest points for each single-class gallery set in SANP and RNP ignores collaboration and competition with other classes, which may cause a wrong-class gallery set to have a small between-set distance. CRNP used collaborative representation to overcome this shortcoming but it doesn't explicitly minimize the between-set distance. In order to solve these issues and fully exploit the advantages of nearest points based approaches, in this paper a novel joint regularized nearest points (JRNP) is proposed for face recognition based on image sets. In JRNP, the nearest point in the query set is generated by considering the entire gallery set of all classes; at the same time, JRNP explicitly minimizes the between-set distance of the query set and a single-class gallery set. Furthermore, we proposed algorithms of greedy JRNP and adaptive JRNP to solve the presented model, and the classification is then based on the joint distance between the regularized nearest points in image sets. Extensive experiments were conducted on benchmark databases (e.g., Honda/UCSD, CMU Mobo, You Tube Celebrities databases, and the large-scale You Tube Face datasets). The experimental results clearly show that our JRNP leads the performance in face recognition based on image sets. We proposed a greedy JRNP (GJRNP) without weight estimation needed in adaptive JRNP.GJRNP considers both collaboration and between-class discrimination of gallery sets.The query-set nearest points of GJRNP flexibly change for different gallery sets.Experiments on large-scale You Tube faces and 47 celebrities datasets are conducted.CMC curves, statistical significance test, and sensitivity analysis are added.

[1]  Shiguang Shan,et al.  Joint sparse representation for video-based face recognition , 2014, Neurocomputing.

[2]  Lei Zhang,et al.  Face recognition based on regularized nearest points between image sets , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[3]  Tsuhan Chen,et al.  Video-based face recognition using adaptive hidden Markov models , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[4]  Ken-ichi Maeda,et al.  Face recognition using temporal image sequence , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[5]  Jun Wang,et al.  Non-uniform patch based face recognition via 2D-DWT , 2015, Image Vis. Comput..

[6]  Lei Zhang,et al.  Sparse representation or collaborative representation: Which helps face recognition? , 2011, 2011 International Conference on Computer Vision.

[7]  Larry S. Davis,et al.  Covariance discriminative learning: A natural and efficient approach to image set classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Allen Y. Yang,et al.  Fast L1-Minimization Algorithms For Robust Face Recognition , 2010, 1007.3753.

[9]  Karim Faez,et al.  Unrestricted pose-invariant face recognition by sparse dictionary matrix , 2015, Image Vis. Comput..

[10]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[12]  Josef Kittler,et al.  Incremental Learning of Locally Orthogonal Subspaces for Set-based Object Recognition , 2006, BMVC.

[13]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[14]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Andrew W. Fitzgibbon,et al.  Joint manifold distance: a new approach to appearance based clustering , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  Hakan Cevikalp,et al.  Face recognition based on image sets , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Shiguang Shan,et al.  Image sets alignment for Video-Based Face Recognition , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Zhifeng Li,et al.  Spatio-temporal Embedding for Statistical Face Recognition from Video , 2006, ECCV.

[19]  Gang Wang,et al.  Image Set Classification Using Holistic Multiple Order Statistics Features and Localized Multi-kernel Metric Learning , 2013, 2013 IEEE International Conference on Computer Vision.

[20]  Dit-Yan Yeung,et al.  Locally Linear Models on Face Appearance Manifolds with Application to Dual-Subspace Based Classification , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[21]  Ralph Gross,et al.  The CMU Motion of Body (MoBo) Database , 2001 .

[22]  Trevor Darrell,et al.  Face Recognition from Long-Term Observations , 2002, ECCV.

[23]  Vladimir Pavlovic,et al.  Face tracking and recognition with visual constraints in real-world videos , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Ruiping Wang,et al.  Manifold Discriminant Analysis , 2009, CVPR.

[25]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[26]  Masashi Nishiyama,et al.  Recognizing Faces of Moving People by Hierarchical Image-Set Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Masayuki Mukunoki,et al.  Collaboratively Regularized Nearest Points for Set Based Recognition , 2013, BMVC.

[28]  David Zhang,et al.  Collaborative Representation based Classification for Face Recognition , 2012, ArXiv.

[29]  Rama Chellappa,et al.  Probabilistic Human Recognition from Video , 2002, ECCV.

[30]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[31]  Josef Kittler,et al.  Discriminative Learning and Recognition of Image Set Classes Using Canonical Correlations , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  LinLin Shen,et al.  Joint regularized nearest points for image set based face recognition , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[33]  Ajmal S. Mian,et al.  Face Recognition Using Sparse Approximated Nearest Points between Image Sets , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Lei Zhang,et al.  Sparsely encoded local descriptor for face verification , 2015, Neurocomputing.

[35]  Masashi Nishiyama,et al.  Face Recognition with the Multiple Constrained Mutual Subspace Method , 2003, AVBPA.

[36]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[37]  Trevor Darrell,et al.  Face recognition with image sets using manifold density divergence , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[39]  Pengfei Shi,et al.  Kernel Grassmannian distances and discriminant analysis for face recognition from image sets , 2009, Pattern Recognit. Lett..

[40]  Yang Gao,et al.  Local histogram specification for face recognition under varying lighting conditions , 2014, Image Vis. Comput..

[41]  Mohammed Bennamoun,et al.  Linear Regression for Face Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[43]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[44]  Simon C. K. Shiu,et al.  Image Set-Based Collaborative Representation for Face Recognition , 2013, IEEE Transactions on Information Forensics and Security.

[45]  Osamu Yamaguchi,et al.  The Kernel Orthogonal Mutual Subspace Method and Its Application to 3D Object Recognition , 2007, ACCV.

[46]  Shengcai Liao,et al.  Partial Face Recognition: Alignment-Free Approach , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Johannes Stallkamp,et al.  Video-based Face Recognition on Real-World Data , 2007, 2007 IEEE 11th International Conference on Computer Vision.