Incremental Linear Discriminant Analysis: A Fast Algorithm and Comparisons

It has always been a challenging task to develop a fast and an efficient incremental linear discriminant analysis (ILDA) algorithm. For this purpose, we conduct a new study for linear discriminant analysis (LDA) in this paper and develop a new ILDA algorithm. We propose a new batch LDA algorithm called LDA/QR. LDA/QR is a simple and fast LDA algorithm, which is obtained by computing the economic QR factorization of the data matrix followed by solving a lower triangular linear system. The relationship between LDA/QR and uncorrelated LDA (ULDA) is also revealed. Based on LDA/QR, we develop a new incremental LDA algorithm called ILDA/QR. The main features of our ILDA/QR include that: 1) it can easily handle the update from one new sample or a chunk of new samples; 2) it has efficient computational complexity and space complexity; and 3) it is very fast and always achieves competitive classification accuracy compared with ULDA algorithm and existing ILDA algorithms. Numerical experiments based on some real-world data sets demonstrate that our ILDA/QR is very efficient and competitive with the state-of-the-art ILDA algorithms in terms of classification accuracy, computational complexity, and space complexity.

[1]  Aleix M. Martínez,et al.  Kernel Optimization in Discriminant Analysis , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[3]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[4]  T. Poggio,et al.  Prediction of central nervous system embryonal tumour outcome based on gene expression , 2002, Nature.

[5]  Jieping Ye,et al.  Least squares linear discriminant analysis , 2007, ICML '07.

[6]  Aleix M. Martínez,et al.  Subclass discriminant analysis , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Sets and Its Applications , 2010, International Journal of Computer Vision.

[8]  Vipin Kumar,et al.  Partitioning-based clustering for Web document categorization , 1999, Decis. Support Syst..

[9]  Yeung Sam Hung,et al.  Characterization of All Solutions for Undersampled Uncorrelated Linear Discriminant Analysis Problems , 2011, SIAM J. Matrix Anal. Appl..

[10]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Set Approximations , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Vasant Honavar,et al.  Learn++: an incremental learning algorithm for supervised neural networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[12]  Vipin Kumar,et al.  WebACE: a Web agent for document categorization and exploration , 1998, AGENTS '98.

[13]  F. Burk,et al.  Lebesgue Measure and Integration: An Introduction , 1997 .

[14]  Peter Bühlmann,et al.  Supervised clustering of genes , 2002, Genome Biology.

[15]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Yiannis Kompatsiaris,et al.  Mixture Subclass Discriminant Analysis Link to Restricted Gaussian Model and Other Generalizations , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[18]  Ash A. Alizadeh,et al.  Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling , 2000, Nature.

[19]  Jieping Ye,et al.  Feature Reduction via Generalized Uncorrelated Linear Discriminant Analysis , 2006, IEEE Transactions on Knowledge and Data Engineering.

[20]  Yong Wang,et al.  Incremental learning of complete linear discriminant analysis for face recognition , 2012, Knowl. Based Syst..

[21]  Zhi-Hua Zhou,et al.  Least Square Incremental Linear Discriminant Analysis , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[22]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[23]  Makedonka Mitreva,et al.  A vertebrate case study of the quality of assemblies derived from next-generation sequences , 2011, Genome Biology.

[24]  Haesun Park,et al.  Structure Preserving Dimension Reduction for Clustered Text Data Based on the Generalized Singular Value Decomposition , 2003, SIAM J. Matrix Anal. Appl..

[25]  Vipin Kumar,et al.  Document Categorization and Query Generation on the World Wide Web Using WebACE , 1999, Artificial Intelligence Review.

[26]  André Stuhlsatz,et al.  Feature Extraction With Deep Neural Networks by a Generalized Discriminant Analysis , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[27]  David G. Stork,et al.  Pattern Classification , 1973 .

[28]  Zhihua Zhang,et al.  Regularized Discriminant Analysis, Ridge Regression and Beyond , 2010, J. Mach. Learn. Res..

[29]  Delin Chu,et al.  A new and fast implementation for null space based linear discriminant analysis , 2010, Pattern Recognit..

[30]  Jieping Ye,et al.  Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems , 2005, J. Mach. Learn. Res..

[31]  Jian Yang,et al.  Why can LDA be performed in PCA transformed space? , 2003, Pattern Recognit..

[32]  Hui Xiong,et al.  IDR/QR: an incremental dimension reduction algorithm via QR decomposition , 2004, IEEE Transactions on Knowledge and Data Engineering.