Eigen‐analysis of nonlinear PCA with polynomial kernels

There has been growing interest in kernel methods for classification, clustering and dimension reduction. For example, kernel Fisher discriminant analysis, spectral clustering and kernel principal component analysis are widely used in statistical learning and data mining applications. The empirical success of the kernel method is generally attributed to nonlinear feature mapping induced by the kernel, which in turn determines a low dimensional data embedding. It is important to understand the effect of a kernel and its associated kernel parameter(s) on the embedding in relation to data distributions. In this paper, we examine the geometry of the nonlinear embedding for kernel principal component analysis (PCA) when polynomial kernels are used. We carry out eigen-analysis of the polynomial kernel operator associated with data distributions and investigate the effect of the degree of polynomial. The results provide both insights into the geometry of nonlinear data embedding and practical guidelines for choosing an appropriate degree for dimension reduction with polynomial kernels. We further comment on the effect of centering kernels on the spectral property of the polynomial kernel operator. © 2013 Wiley Periodicals, Inc. Statistical Analysis and Data Mining, 2013

[1]  Victor Vianu,et al.  Invited articles section foreword , 2010, JACM.

[2]  Jeongyoun Ahn,et al.  A stable hyperparameter selection for the Gaussian RBF kernel for discrimination , 2010 .

[3]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[4]  D Haussler,et al.  Knowledge-based analysis of microarray gene expression data by using support vector machines. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Johan A. K. Suykens,et al.  Sparse conjugate directions pursuit with application to fixed-size kernel models , 2011, Machine Learning.

[6]  Johan A. K. Suykens,et al.  Optimized fixed-size kernel models for large data sets , 2010, Comput. Stat. Data Anal..

[7]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[8]  Jeongyoun Ahn.  A stable hyperparameter selection for the Gaussian RBF kernel for discrimination , 2010, Stat. Anal. Data Min..

[9]  Linda Kaufman,et al.  Solving the quadratic programming problem arising in support vector classification , 1999 .

[10]  Michael Rabadi,et al.  Kernel Methods for Machine Learning , 2015 .

[11]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[12]  Guy L. Scott,et al.  Feature grouping by 'relocalisation' of eigenvectors of the proximity matrix , 1990, BMVC.

[14]  Christopher K. I. Williams,et al.  Gaussian regression and optimal finite dimensional linear models , 1997 .

[15]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[16]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[17]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[18]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[19]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[20]  G. Wahba.  Spline models for observational data , 1990 .

[21]  Mia Hubert,et al.  ROBPCA: A New Approach to Robust Principal Component Analysis , 2005, Technometrics.

[22]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[23]  G. Baudat,et al.  Generalized Discriminant Analysis Using a Kernel Approach , 2000, Neural Computation.

[24]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[25]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[26]  Noureddine El Karoui,et al.  The spectrum of kernel random matrices , 2010, 1001.0492.

[27]  Mikhail Belkin,et al.  DATA SPECTROSCOPY: EIGENSPACES OF CONVOLUTION OPERATORS AND CLUSTERING , 2008, 0807.3719.