Manifold regularization and semi-supervised learning: some theoretical analyses

Manifold regularization (Belkin et al., 2006) is a geometrically motivated framework for machine learning within which several semi-supervised algorithms have been constructed. Here we try to provide some theoretical understanding of this approach. Our main result is to expose the natural structure of a class of problems on which manifold regularization methods are helpful. We show that for such problems, no supervised learner can learn effectively. On the other hand, a manifold based learner (that knows the manifold or "learns" it from unlabeled examples) can learn with relatively few labeled examples. Our analysis follows a minimax style with an emphasis on finite sample results (in terms of n: the number of labeled examples). These results allow us to properly interpret manifold regularization and related spectral and geometric algorithms in terms of their potential use in semi-supervised learning.

[1]  Stephen Smale,et al.  Finding the Homology of Submanifolds with High Confidence from Random Samples , 2008, Discret. Comput. Geom..

[2]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[3]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[4]  Mikhail Belkin,et al.  Towards a theoretical foundation for Laplacian-based manifold methods , 2005, J. Comput. Syst. Sci..

[5]  P. Bickel,et al.  Local polynomial regression on unknown manifolds , 2007, 0708.0983.

[6]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[7]  Ulrike von Luxburg,et al.  From Graphs to Manifolds - Weak and Strong Pointwise Consistency of Graph Laplacians , 2005, COLT.

[8]  Regina Y. Liu,et al.  Complex datasets and inverse problems : tomography, networks and beyond , 2007, 0708.1130.

[9]  Vikas Sindhwani,et al.  On Manifold Regularization , 2005, AISTATS.

[10]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[11]  Larry A. Wasserman,et al.  Statistical Analysis of Semi-Supervised Regression , 2007, NIPS.

[12]  Mikhail Belkin,et al.  Manifold Regularization : A Geometric Framework for Learning from Examples , 2004 .

[13]  H. Weizsäcker,et al.  Chernoff's Theorem and Discrete Time Approximations of Brownian Motion on Manifolds , 2004, math/0409155.

[14]  Afra Zomorodian,et al.  Computing Persistent Homology , 2004, SCG '04.

[15]  S. Rosenberg.  The Laplacian on a Riemannian Manifold: The Construction of the Heat Kernel , 1997 .

[16]  S. Rosenberg.  The Laplacian on a Riemannian Manifold: The Laplacian on a Riemannian Manifold , 1997 .

[17]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[18]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  H. Weizsäcker,et al.  Brownian motion on a manifold as a limit of stepwise conditioned standard Brownian motions , 2000 .

[20]  Mikhail Belkin,et al.  Convergence of Laplacian Eigenmaps , 2006, NIPS.

[21]  Xiaojin Zhu,et al.  --1 CONTENTS , 2006 .

[22]  V. Koltchinskii,et al.  Empirical graph Laplacian approximation of Laplace–Beltrami operators: Large sample results , 2006, math/0612777.

[23]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[24]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[25]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Vittorio Castelli,et al.  The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter , 1996, IEEE Trans. Inf. Theory.

[27]  Guillermo Sapiro,et al.  Comparing point clouds , 2004, SGP '04.

[28]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[29]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.