Approximate Manifold Regularization: Scalable Algorithm and Generalization Analysis

Graph-based semi-supervised learning is one of the most popular and successful semi-supervised learning approaches. Unfortunately, it suffers from high time and space complexity, at least quadratic with the number of training samples. In this paper, we propose an efficient graph-based semisupervised algorithm with a sound theoretical guarantee. The proposed method combines Nystrom subsampling and preconditioned conjugate gradient descent, substantially improving computational efficiency and reducing memory requirements. Extensive empirical results reveal that our method achieves the state-of-the-art performance in a short time even with limited computing resources.

[1]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[2]  Wei Liu,et al.  Robust and Scalable Graph-Based Semisupervised Learning , 2012, Proceedings of the IEEE.

[3]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[4]  Weiping Wang,et al.  Fast Cross-Validation , 2018, IJCAI.

[5]  Lorenzo Rosasco,et al.  FALKON: An Optimal Large Scale Kernel Method , 2017, NIPS.

[6]  Jose M. Such,et al.  International Joint Conference on Artificial Intelligence (IJCAI) , 2016 .

[7]  Yong Liu,et al.  Efficient Approximation of Cross-Validation for Kernel Methods using Bouligand Influence Function , 2014, ICML.

[8]  Rynson W. H. Lau,et al.  Knowledge and Data Engineering for e-Learning Special Issue of IEEE Transactions on Knowledge and Data Engineering , 2008 .

[9]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[10]  Joachim M. Buhmann,et al.  Correlated random features for fast semi-supervised learning , 2013, NIPS.

[11]  Wei Liu,et al.  Large Graph Construction for Scalable Semi-Supervised Learning , 2010, ICML.

[12]  Thomas G. Dietterich,et al.  In Advances in Neural Information Processing Systems 12 , 1991, NIPS 1991.

[13]  Claudio Gentile,et al.  Random Spanning Trees and the Prediction of Weighted Graphs , 2010, ICML.

[14]  Mikhail Belkin,et al.  Laplacian Support Vector Machines Trained in the Primal , 2009, J. Mach. Learn. Res..

[15]  Lorenzo Rosasco,et al.  Less is More: Nyström Computational Regularization , 2015, NIPS.

[16]  Koby Crammer,et al.  Breaking the curse of kernelization: budgeted stochastic gradient descent for large-scale SVM training , 2012, J. Mach. Learn. Res..

[17]  Yong Liu,et al.  Fast Cross-Validation for Kernel-Based Algorithms , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 34. NO. 4, JULY 1996 Universal Multifractal Scaling of Synthetic , 1996 .

[19]  Ameet Talwalkar,et al.  Large-scale SVD and manifold learning , 2013, J. Mach. Learn. Res..

[20]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[21]  Arieh Iserles,et al.  On the Foundations of Computational Mathematics , 2001 .

[22]  Yong Liu,et al.  Multi-Class Learning: From Theory to Algorithm , 2018, NeurIPS.

[23]  Xiao-Tong Yuan,et al.  Large-Scale Graph-Based Semi-Supervised Learning via Tree Laplacian Solver , 2016, AAAI.

[24]  Abhishake Rastogi,et al.  M L ] 1 3 O ct 2 01 7 Manifold regularization based on Nyström type subsampling , 2018 .