Spectral Connectivity Analysis

Spectral kernel methods are techniques for mapping data into a coordinate system that efficiently reveals the geometric structure—in particular, the “connectivity”—of the data. These methods depend on tuning parameters. We analyze the dependence of the method on these tuning parameters. We focus on one particular technique—diffusion maps—but our analysis can be used for other spectral methods as well. We identify the key population quantities, we define an appropriate risk function for analyzing the estimators, and we explain how these methods relate to classical kernel smoothing. We also show that, in some cases, fast rates of convergence are possible even in high dimensions. The Appendix of the article is available online as supplementary materials.

[1]  Nicolas Le Roux,et al.  Learning Eigenfunctions Links Spectral Embedding and Kernel PCA , 2004, Neural Computation.

[2]  Larry A. Wasserman,et al.  Statistical Analysis of Semi-Supervised Regression , 2007, NIPS.

[3]  L. Sodré,et al.  Semi‐empirical analysis of Sloan Digital Sky Survey galaxies – I. Spectral synthesis method , 2005 .

[4]  M. Mackey,et al.  Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics , 1998 .

[5]  Peter J. Bickel,et al.  Maximum Likelihood Estimation of Intrinsic Dimension , 2004, NIPS.

[6]  Robert D. Nowak,et al.  Unlabeled data: Now it helps, now it doesn't , 2008, NIPS.

[7]  Shai Ben-David,et al.  A Sober Look at Clustering Stability , 2006, COLT.

[8]  T. Hastie,et al.  Principal Curves , 2007 .

[9]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[10]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: multiscale methods. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Laurence A. Baxter,et al.  Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics, Second Edition , 1994, by Andrzej Lasota and Michael C. Mackey, Applied Mathematical Sciences, vol. 97 (New York: Springer-Verlag), xiv + 472 pp. , 1996, Probability in the Engineering and Informational Sciences.

[12]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[13]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Ann B. Lee,et al.  EXPLOITING LOW-DIMENSIONAL STRUCTURE IN ASTRONOMICAL SPECTRA , 2008, 0807.2900.

[16]  Helge J. Ritter,et al.  Quantizing Density Estimators , 2001, NIPS.

[17]  Gilles Blanchard,et al.  On the Convergence of Eigenspaces in Kernel Principal Component Analysis , 2005, NIPS.

[18]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[19]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[20]  Mikhail Belkin,et al.  DATA SPECTROSCOPY: EIGENSPACES OF CONVOLUTION OPERATORS AND CLUSTERING , 2008, 0807.3719.

[21]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[22]  G. Stewart Perturbation theory for the singular value decomposition , 1990 .

[23]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[24]  Arthur D. Szlam,et al.  Diffusion wavelet packets , 2006 .

[25]  V. Koltchinskii,et al.  Empirical graph Laplacian approximation of Laplace–Beltrami operators: Large sample results , 2006, math/0612777.

[26]  Mikhail Belkin,et al.  Consistency of spectral clustering , 2008, 0804.0678.

[27]  Nanda Kambhatla,et al.  Dimension Reduction by Local Principal Component Analysis , 1997, Neural Computation.

[28]  Adam Krzyzak,et al.  On the Rate of Convergence of Local Averaging Plug-In Classification Rules Under a Margin Condition , 2007, IEEE Transactions on Information Theory.

[29]  Jianqing Fan Local Linear Regression Smoothers and Their Minimax Efficiencies , 1993 .

[30]  François Fouss,et al.  A novel way of computing similarities between nodes of a graph, with application to collaborative recommendation , 2005, The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI'05).

[31]  F. Scholz Maximum Likelihood Estimation , 2006 .

[32]  Mikhail Belkin,et al.  Semi-Supervised Learning on Riemannian Manifolds , 2004, Machine Learning.

[33]  Tommi S. Jaakkola,et al.  Partially labeled classification with Markov random walks , 2001, NIPS.

[34]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Ulrike von Luxburg,et al.  From Graphs to Manifolds - Weak and Strong Pointwise Consistency of Graph Laplacians , 2005, COLT.

[36]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[37]  Matthias Hein,et al.  Measure Based Regularization , 2003, NIPS.

[38]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[39]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[40]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[41]  Michel Verleysen,et al.  Nonlinear Dimensionality Reduction , 2021, Computer Vision.

[42]  Luc De Raedt,et al.  Proceedings of the 22nd international conference on Machine learning , 2005 .

[43]  P. Murdin MONTHLY NOTICES OF THE ROYAL ASTRONOMICAL SOCIETY , 2005 .

[44]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[45]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[46]  A. Tsybakov,et al.  Fast learning rates for plug-in classifiers , 2007, 0708.2321.

[47]  Astronomy,et al.  Photometric Redshift Estimation Using Spectral Connectivity Analysis , 2009, 0906.0995.

[48]  Chad M. Schafer,et al.  High-Dimensional Density Estimation via SCA: An Example in the Modelling of Hurricane Tracks ✩ , 2009, 0907.0199.

[49]  J. P. Torres-Papaqui,et al.  The history of star-forming galaxies in the Sloan Digital Sky Survey , 2007, 0707.3578.

[50]  Mikhail Belkin,et al.  Towards a Theoretical Foundation for Laplacian-Based Manifold Methods , 2005, COLT.

[51]  E. Giné,et al.  Rates of strong uniform consistency for multivariate kernel density estimators , 2002 .

[52]  A. Singer From graph to manifold Laplacian: The convergence rate , 2006 .

[53]  D. Donoho,et al.  Hessian Eigenmaps : new locally linear embedding techniques for high-dimensional data , 2003 .

[54]  Ann B. Lee,et al.  Accurate parameter estimation for star formation history in galaxies using SDSS spectra , 2009, 0905.4683.

[55]  R. Coifman,et al.  Diffusion Wavelets , 2004 .

[56]  Joachim M. Buhmann,et al.  Stability-Based Validation of Clustering Solutions , 2004, Neural Computation.

[57]  E. Mammen,et al.  Smooth Discrimination Analysis , 1999 .

[58]  Ronald R. Coifman,et al.  Graph Laplacian Tomography From Unknown Random Projections , 2008, IEEE Transactions on Image Processing.