On incremental semi-supervised discriminant analysis

In various pattern classification problems, semi-supervised discriminant analysis has shown its effectiveness in utilizing unlabeled data to yield better performance than linear discriminant analysis. However, many of these semi-supervised classifiers operate in batch-mode and do not allow to incrementally update the existing model, which is one of the major limitations. This paper presents an incremental semi-supervised discriminant analysis algorithm, which utilizes the unlabeled data for enabling incremental learning. The major contributions of this research are (1) utilizing large unlabeled training set to estimate the total scatter matrix, (2) incremental learning approach that requires updating only the between-class scatter matrix and not the total scatter matrix, and (3) utilizing manifold regularization for robust estimation of total variability and sufficient spanning set representation for incremental learning. Using face recognition as the case study, evaluation is performed on the CMU-PIE, CMU-MultiPIE, and NIR-VIS-2.0 datasets. The experimental results show that the incremental model is consistent with the batch counterpart and reduces the training time significantly. HighlightsIncremental semi-supervised discriminant analysis algorithm is proposed.Large unlabeled data is utilized to estimate total scatter in discriminant analysis.It does not require to incrementally update the total scatter eigenmodel.A face recognition case study is shown on CMU-PIE, NIR-VIS-2.0, and MultiPIE databases.The proposed ISSDA requires significantly less computational time and maintains accuracy.

[1]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Sets and Its Applications , 2010, International Journal of Computer Vision.

[2]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[3]  Feiping Nie,et al.  Semi-supervised sub-manifold discriminant analysis , 2008, Pattern Recognit. Lett..

[4]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[5]  Wei Liang,et al.  Incremental discriminant-analysis of canonical correlations for action recognition , 2010, Pattern Recognit..

[6]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Set Approximations , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Shaoning Pang,et al.  Incremental linear discriminant analysis for classification of data streams , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[8]  Wei Jia,et al.  Locality preserving discriminant projections for face and palmprint recognition , 2010, Neurocomputing.

[9]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[10]  Xian-Sheng Hua,et al.  Ensemble Manifold Regularization , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Dacheng Tao,et al.  Double Shrinking Sparse Dimension Reduction , 2013, IEEE Transactions on Image Processing.

[12]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[13]  Juyang Weng,et al.  Using Discriminant Eigenfeatures for Image Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Jiawei Han,et al.  Orthogonal Laplacianfaces for Face Recognition , 2006, IEEE Transactions on Image Processing.

[15]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Yong Wang,et al.  Incremental complete LDA for face recognition , 2012, Pattern Recognit..

[17]  Pong C. Yuen,et al.  Incremental Linear Discriminant Analysis for Face Recognition , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[18]  Shengcai Liao,et al.  The CASIA NIR-VIS 2.0 Face Database , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[19]  Hui Xiong,et al.  IDR/QR: An Incremental Dimension Reduction Algorithm via QR Decomposition , 2005, IEEE Trans. Knowl. Data Eng..

[20]  Tao Jiang,et al.  Efficient and robust feature extraction by maximum margin criterion , 2003, IEEE Transactions on Neural Networks.

[21]  J. McClellan,et al.  On discriminative semi-supervised incremental learning with a multi-view perspective for image concept modeling , 2012 .

[22]  Jie Gui,et al.  Multi-step dimensionality reduction and semi-supervised graph-based tumor classification using gene expression data , 2010, Artif. Intell. Medicine.

[23]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[24]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[25]  Cheng-Lin Liu,et al.  Evaluation of weighted Fisher criteria for large category dimensionality reduction in application to Chinese handwriting recognition , 2013, Pattern Recognit..

[26]  Yong Wang,et al.  Incremental learning from chunk data for IDR/QR , 2015, Image Vis. Comput..

[27]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[28]  Shinichi Nakajima,et al.  Semi-supervised local Fisher discriminant analysis for dimensionality reduction , 2009, Machine Learning.

[29]  Jiawei Han,et al.  Semi-supervised Discriminant Analysis , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[30]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[31]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[32]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[33]  J KriegmanDavid,et al.  Eigenfaces vs. Fisherfaces , 1997 .

[34]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[35]  Zhi-Hua Zhou,et al.  Least Square Incremental Linear Discriminant Analysis , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[36]  I. Jolliffe.  Principal Component Analysis , 2002 .

[37]  Aleix M. Martínez,et al.  Subclass discriminant analysis , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Xiaofei He.  Incremental semi-supervised subspace learning for image retrieval , 2004, MULTIMEDIA '04.

[39]  Yong Wang,et al.  Incremental learning of complete linear discriminant analysis for face recognition , 2012, Knowl. Based Syst..

[40]  Jieping Ye,et al.  An optimization criterion for generalized discriminant analysis on undersampled problems , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[42]  Dit-Yan Yeung,et al.  Semisupervised Generalized Discriminant Analysis , 2011, IEEE Transactions on Neural Networks.

[43]  Hwann-Tzong Chen,et al.  Local discriminant embedding and its variants , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[44]  Ralph R. Martin,et al.  Merging and Splitting Eigenspace Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[46]  Richa Singh,et al.  Incremental subclass discriminant analysis: A case study in face recognition , 2012, 2012 19th IEEE International Conference on Image Processing.

[47]  Wei Jia,et al.  Discriminant sparse neighborhood preserving embedding for face recognition , 2012, Pattern Recognit..

[48]  Yu-Chiang Frank Wang,et al.  A rank-one update method for least squares linear discriminant analysis with concept drift , 2013, Pattern Recognit..

[49]  Michael K. Ng,et al.  Incremental Linear Discriminant Analysis: A Fast Algorithm and Comparisons , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[50]  Minho Lee,et al.  Extension of Incremental Linear Discriminant Analysis to Online Feature Extraction under Nonstationary Environments , 2012, ICONIP.