A Note on Learning with Integral Operators

A large number of learning algorithms, for example, spectral clustering, kernel Principal Components Analysis and many manifold methods, are based on estimating eigenvalues and eigenfunctions of operators defined by a similarity function or a kernel, given empirical data. Thus for the analysis of algorithms, it is an important problem to be able to assess the quality of such approximations. The contribution of our paper is two-fold: 1. We use a technique based on a concentration inequality for Hilbert spaces to provide new much simplified proofs for a number of results in spectral approximation. 2. Using these methods we provide several new results for estimating spectral properties of the graph Laplacian operator extending and strengthening results from [27].

[1]  S. Mendelson,et al.  On singular values of matrices with independent rows , 2006 .

[2]  Nello Cristianini,et al.  On the Concentration of Spectral Properties , 2001, NIPS.

[3]  V. Burenkov.  Sobolev spaces on domains , 1998 .

[4]  P. Anselone,et al.  Collectively Compact Operator Approximation Theory and Applications to Integral Equations , 1971 .

[5]  Matthias Hein,et al.  Uniform Convergence of Adaptive Graph-Based Regularization , 2006, COLT.

[6]  Lorenzo Rosasco,et al.  On regularization algorithms in learning theory , 2007, J. Complex..

[7]  Nello Cristianini,et al.  On the eigenspectrum of the gram matrix and the generalization error of kernel-PCA , 2005, IEEE Transactions on Information Theory.

[8]  S. Smale,et al.  Geometry on Probability Spaces , 2009 .

[9]  V. Koltchinskii,et al.  Random matrix approximation of spectra of integral operators , 2000 .

[10]  Tosio Kato.  Perturbation theory for linear operators , 1966 .

[11]  V. Koltchinskii.  Asymptotics of Spectral Projections of Some Random Matrices Approximating Integral Operators , 1998 .

[12]  A. Singer.  From graph to manifold Laplacian: The convergence rate , 2006 .

[13]  Mikhail Belkin,et al.  A Note on Perturbation Results for Learning Empirical Operators , 2008 .

[14]  Mikhail Belkin,et al.  Towards a theoretical foundation for Laplacian-based manifold methods , 2005, J. Comput. Syst. Sci..

[15]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[16]  Mikhail Belkin,et al.  Consistency of spectral clustering , 2008, 0804.0678.

[17]  Ulrike von Luxburg,et al.  On the Convergence of Spectral Clustering on Random Samples: The Normalized Case , 2004, COLT.

[18]  V. Koltchinskii,et al.  Empirical graph Laplacian approximation of Laplace–Beltrami operators: Large sample results , 2006, math/0612777.

[19]  Gilles Blanchard,et al.  Statistical properties of kernel principal component analysis , 2007, Machine Learning.

[20]  Steven W. Zucker,et al.  Diffusion Maps and Geometric Harmonics for Automatic Target Recognition (ATR). Volume 2. Appendices , 2007 .

[21]  Tosio Kato.  Variation of discrete spectra , 1987 .

[22]  S. Lang.  Real and Functional Analysis , 1983 .

[23]  I. Pinelis.  AN APPROACH TO INEQUALITIES FOR THE DISTRIBUTIONS OF INFINITE-DIMENSIONAL MARTINGALES , 1992 .

[24]  Ulrike von Luxburg,et al.  From Graphs to Manifolds - Weak and Strong Pointwise Consistency of Graph Laplacians , 2005, COLT.

[25]  Gilles Blanchard,et al.  On the Convergence of Eigenspaces in Kernel Principal Component Analysis , 2005, NIPS.

[26]  Lorenzo Rosasco,et al.  Learning from Examples as an Inverse Problem , 2005, J. Mach. Learn. Res..

[27]  Mikhail Belkin,et al.  Convergence of Laplacian Eigenmaps , 2006, NIPS.

[28]  Shahar Mendelson,et al.  Ellipsoid Approximation Using Random Vectors , 2005, COLT.