Fast incremental LDA feature extraction

Linear discriminant analysis (LDA) is a traditional statistical technique that reduces dimensionality while preserving as much of the class discriminatory information as possible. The conventional form of the LDA assumes that all the data are available in advance and the LDA feature space is computed by finding the eigendecomposition of an appropriate matrix. However, there are situations where the data are presented in a sequence and the LDA features are required to be updated incrementally by observing the new incoming samples. Chatterjee and Roychowdhury proposed an algorithm for incrementally computing the LDA features followed by Moghaddam et al. who accelerated the convergence rate of these algorithms. The proposed algorithms by Moghaddam et al. are derived by applying the chain rule on an implicit cost function. Since the authors have not had access to the cost function they could not analyze the convergence of the proposed algorithms and the convergence of the proposed accelerated techniques were not guaranteed. In this paper, we briefly review the previously proposed algorithms, then we derive new algorithms to accelerate the convergence rate of the incremental LDA algorithm given by Chatterjee and Roychowdhury. The proposed algorithms are derived by optimizing the step size in each iteration using steepest descent and conjugate direction methods. We test the performance of the proposed algorithms for incremental LDA on synthetic and real data sets. The simulation results confirm that the proposed algorithms estimate the LDA features faster than the gradient descent based algorithm presented by Moghaddam et al., and the algorithm proposed by Chatterjee and Roychowdhury. HighlightsThe previous algorithms for fast incremental LDA are discussed.New algorithm for fast incremental LDA based on steepest descent is proposed.New algorithm for fast incremental LDA based on conjugate direction is proposed.The performance of the proposed algorithms is tested using real data sets.

[1]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Set Approximations , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  C. Murukesh,et al.  Intellectually combined face recognition using curvelet based principle component analysis for feature extraction and Bayesian Classifier , 2011, 2011 International Conference on Signal Processing, Communication, Computing and Networking Technologies.

[4]  Shingo Tomita,et al.  An optimal orthonormal system for discriminant analysis , 1985, Pattern Recognit..

[5]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[6]  Jiawei Han,et al.  Orthogonal Laplacianfaces for Face Recognition , 2006, IEEE Transactions on Image Processing.

[7]  Hui Xiong,et al.  IDR/QR: an incremental dimension reduction algorithm via QR decomposition , 2004, IEEE Transactions on Knowledge and Data Engineering.

[8]  Shaoning Pang,et al.  Incremental learning of feature space and classifier for on-line pattern recognition , 2006, Int. J. Knowl. Based Intell. Eng. Syst..

[9]  Yongmin Li,et al.  On incremental and robust subspace learning , 2004, Pattern Recognit..

[10]  Pierre Priouret,et al.  Adaptive Algorithms and Stochastic Approximations , 1990, Applications of Mathematics.

[11]  R. A. Silverman,et al.  Introductory Real Analysis , 1972 .

[12]  M. Ringnér,et al.  Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks , 2001, Nature Medicine.

[13]  Hamid Abrishami Moghaddam,et al.  Fast adaptive LDA using quasi-Newton algorithm , 2007, Pattern Recognit. Lett..

[14]  Yuxiao Hu,et al.  Learning a Spatially Smooth Subspace for Face Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  NenadicZoran,et al.  Approximate information discriminant analysis , 2008 .

[16]  Martin T. Hagan,et al.  Neural network design , 1995 .

[17]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[18]  Erkki Oja,et al.  Principal components, minor components, and linear neural networks , 1992, Neural Networks.

[19]  Juyang Weng,et al.  Candid Covariance-Free Incremental Principal Component Analysis , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Xin Yao,et al.  Linear dimensionality reduction using relevance weighted LDA , 2005, Pattern Recognit..

[21]  Horst Bischof,et al.  Incremental LDA Learning by Combining Reconstructive and Discriminative Approaches , 2007, BMVC.

[22]  佐藤 保,et al.  Principal Components , 2021, Encyclopedic Dictionary of Archaeology.

[23]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[24]  Kuldip K. Paliwal,et al.  Class-dependent PCA, MDC and LDA: A combined classifier for pattern classification , 2006, Pattern Recognit..

[25]  Gene H. Golub,et al.  Matrix computations , 1983 .

[26]  Y.A. Ghassabeh,et al.  A New Incremental Face Recognition System , 2007, 2007 4th IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications.

[27]  A. Berman,et al.  Completely Positive Matrices , 2003 .

[28]  Lianwen Jin,et al.  Incremental learning of LDA model for Chinese writer adaptation , 2010, Neurocomputing.

[29]  SharmaAlok,et al.  Class-dependent PCA, MDC and LDA , 2006 .

[30]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[31]  Vwani P. Roychowdhury,et al.  On self-organizing algorithms and networks for class-separability features , 1997, IEEE Trans. Neural Networks.

[32]  Hamid Abrishami Moghaddam,et al.  Fast adaptive algorithms and networks for class-separability features , 2003, Pattern Recognit..

[33]  Shiguang Shan,et al.  Adaptive discriminant learning for face recognition , 2013, Pattern Recognit..

[34]  Shaoning Pang,et al.  Incremental linear discriminant analysis for classification of data streams , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Güleser Kalayci Demir,et al.  Online local learning algorithms for linear discriminant analysis , 2005, Pattern Recognit. Lett..

[36]  Ja-Chen Lin,et al.  A new LDA-based face recognition system which can solve the small sample size problem , 1998, Pattern Recognit..

[37]  Jasbir S. Arora,et al.  Introduction to Optimum Design , 1988 .

[38]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[39]  Hamid Abrishami Moghaddam,et al.  Algorithms and networks for accelerated convergence of adaptive LDA , 2005, Pattern Recognit..

[40]  Josef Kittler,et al.  Incremental Linear Discriminant Analysis Using Sufficient Spanning Sets and Its Applications , 2010, International Journal of Computer Vision.

[41]  David G. Luenberger,et al.  Linear and Nonlinear Programming: Second Edition , 2003 .

[42]  David G. Stork,et al.  Pattern Classification , 1973 .

[43]  Hamid Abrishami Moghaddam,et al.  Adaptive linear discriminant analysis for online feature extraction , 2013, Machine Vision and Applications.

[44]  Jasbir S Arora,et al.  Introduction to Optimum Design (Third Edition) , 2012 .

[45]  Zoran Nenadic,et al.  Approximate information discriminant analysis: A computationally simple heteroscedastic feature extraction technique , 2008, Pattern Recognit..

[46]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.