Two-Phase Incremental Kernel PCA for Learning Massive or Online Datasets

As a powerful nonlinear feature extractor, kernel principal component analysis (KPCA) has been widely adopted in many machine learning applications. However, KPCA is usually performed in a batch mode, leading to some potential problems when handling massive or online datasets. To overcome this drawback of KPCA, in this paper, we propose a two-phase incremental KPCA (TP-IKPCA) algorithm which can incorporate data into KPCA in an incremental fashion. In the first phase, an incremental algorithm is developed to explicitly express the data in the kernel space. In the second phase, we extend an incremental principal component analysis (IPCA) to estimate the kernel principal components. Extensive experimental results on both synthesized and real datasets showed that the proposed TP-IKPCA produces similar principal components as conventional batch-based KPCA but is computationally faster than KPCA and its several incremental variants. Therefore, our algorithm can be applied to massive or online datasets where the batch method is not available.

[1]  S. V. N. Vishwanathan,et al.  Fast Iterative Kernel Principal Component Analysis , 2007, J. Mach. Learn. Res..

[2]  Yongmin Li,et al.  On incremental and robust subspace learning , 2004, Pattern Recognit..

[3]  Shigeo Abe,et al.  An Efficient Incremental Kernel Principal Component Analysis for Online Feature Selection , 2007, 2007 International Joint Conference on Neural Networks.

[4]  Heeyoung Kim,et al.  Application of kernel principal component analysis to multi-characteristic parameter design problems , 2015, Annals of Operations Research.

[5]  Yin Yang,et al.  Multi-beam pattern synthesis algorithm based on kernel principal component analysis and semi-definite relaxation , 2018, IET Commun..

[6]  Fredrik Hallgren,et al.  Incremental kernel PCA and the Nyström method , 2018, ArXiv.

[7]  Byung-Joo Kim,et al.  Incremental Nonlinear PCA for Classification , 2004, PKDD.

[8]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[9]  Ralph R. Martin,et al.  Incremental Eigenanalysis for Classification , 1998, BMVC.

[10]  G. Baudat,et al.  Kernel-based methods and function approximation , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[11]  Shigeo Abe,et al.  Incremental Kernel PCA for Online Learning of Feature Space , 2005, International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC'06).

[12]  E. Oja.  Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[13]  Tat-Jun Chin,et al.  Incremental Kernel Principal Component Analysis , 2007, IEEE Transactions on Image Processing.

[14]  Xinpeng Zhang,et al.  Kernel quaternion principal component analysis and its application in RGB-D object recognition , 2017, Neurocomputing.

[15]  Tat-Jun Chin,et al.  Incremental Kernel PCA for Efficient Non-linear Feature Extraction , 2006, BMVC.

[16]  Roman Rosipal,et al.  An Expectation-Maximization Approach to Nonlinear Component Analysis , 2001, Neural Computation.

[17]  Sandro Nicole,et al.  Feedforward neural networks for principal components extraction , 2000 .

[18]  Lei Wang,et al.  Modified kernel principal component analysis using double-weighted local outlier factor and its application to nonlinear process monitoring. , 2017, ISA transactions.

[19]  W. Marsden.  I and J , 2012 .

[20]  Seiichi Ozawa,et al.  A fast incremental Kernel Principal Component Analysis for learning stream of data chunks , 2011, The 2011 International Joint Conference on Neural Networks.

[21]  Seiichi Ozawa,et al.  Online feature extraction based on accelerated kernel principal component analysis for data stream , 2015, Evolving Systems.

[22]  Byung-Joo Kim.  Active Visual Learning and Recognition Using Incremental Kernel PCA , 2005, Australian Conference on Artificial Intelligence.

[23]  Haitao Zhao,et al.  A novel incremental principal component analysis and its application for face recognition , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[24]  Kurt Stadlthanner,et al.  KPCA denoising and the pre-image problem revisited , 2008, Digit. Signal Process..

[25]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[26]  Tongsong Jiang,et al.  An Algorithm for Coneigenvalues and Coneigenvectors of Quaternion Matrices , 2015 .

[27]  Bernhard Schölkopf,et al.  Iterative kernel principal component analysis for image modeling , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Juyang Weng,et al.  Candid Covariance-Free Incremental Principal Component Analysis , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Yongli Wang,et al.  On the convergence of asynchronous parallel algorithm for large-scale linearly constrained minimization problem , 2009, Appl. Math. Comput..

[30]  Seiichi Ozawa,et al.  A fast Incremental Kernel Principal Component Analysis for data streams , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[31]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[32]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[33]  Wenming Zheng,et al.  An Improved Algorithm for Kernel Principal Component Analysis , 2005, Neural Processing Letters.

[34]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[35]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[36]  Václav Hlavác,et al.  Greedy Algorithm for a Training Set Reduction in the Kernel Methods , 2003, CAIP.