Statistical Consistency of Kernel Canonical Correlation Analysis

While kernel canonical correlation analysis (CCA) has been applied in many contexts, the convergence of finite sample estimates of the associated functions to their population counterparts has not yet been established. This paper gives a mathematical proof of the statistical convergence of kernel CCA, providing a theoretical justification for the method. The proof uses covariance operators defined on reproducing kernel Hilbert spaces, and analyzes the convergence of their empirical estimates of finite rank to their population counterparts, which can have infinite rank. The result also gives a sufficient condition for convergence on the regularization coefficient involved in kernel CCA: this should decrease as n-1/3, where n is the number of data.

[1]  C. Baker.  Joint measures and cross-covariance operators , 1973 .

[2]  A. V. D. Vaart,et al.  Asymptotic Statistics: U -Statistics , 1998 .

[3]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[4]  Bernhard Schölkopf,et al.  Measuring Statistical Dependence with Hilbert-Schmidt Norms , 2005, ALT.

[5]  Gilles Blanchard,et al.  On the Convergence of Eigenspaces in Kernel Principal Component Analysis , 2005, NIPS.

[6]  Bernhard Schölkopf,et al.  Kernel Constrained Covariance for Dependence Measurement , 2005, AISTATS.

[7]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[8]  N. Vakhania,et al.  Probability Distributions on Banach Spaces , 1987 .

[9]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[10]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[11]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[12]  Michael I. Jordan,et al.  Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces , 2004, J. Mach. Learn. Res..

[13]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation. , 1985 .

[14]  David R. Hardoon,et al.  KCCA for fMRI Analysis , 2004 .

[15]  Yoshihiro Yamanishi,et al.  Extraction of correlated gene clusters from multiple genomic data by generalized kernel canonical correlation analysis , 2003, ISMB.

[16]  Michel Loève,et al.  Probability Theory I , 1977 .

[17]  Kazuyuki Aihara,et al.  Detecting Hidden Synchronization of Chaotic Dynamical Systems: A Kernel-based Approach , 2005 .

[18]  彰 五十嵐.  N. Dunford and J. T. Schwartz (with the assistance of W. G. Bade and R. G. Bartle): Linear Operators. : Part II. Spectral Theoty. Self Adjoint Operators in Hilbert Space. Interscience. 1963. X+1065+7頁, 16×23.5cm, 14,000円。 , 1964 .

[19]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[20]  C. W. Groetsch,et al.  The theory of Tikhonov regularization for Fredholm equations of the first kind , 1984 .

[21]  R. Clarke,et al.  Theory and Applications of Correspondence Analysis , 1985 .

[22]  Shotaro Akaho,et al.  A kernel method for canonical correlation analysis , 2006, ArXiv.

[23]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[24]  Jane-Ling Wang,et al.  Functional canonical analysis for square integrable stochastic processes , 2003 .

[25]  Horst Bischof,et al.  Nonlinear Feature Extraction Using Generalized Canonical Correlation Analysis , 2001, ICANN.

[26]  Don R. Hush,et al.  An Explicit Description of the Reproducing Kernel Hilbert Spaces of Gaussian RBF Kernels , 2006, IEEE Transactions on Information Theory.

[27]  A. Buja.  Remarks on Functional Canonical Variates, Alternating Least Squares Methods and Ace , 1990 .

[28]  B. Silverman,et al.  Canonical correlation analysis when the data are curves. , 1993 .

[29]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003 .