Optimal Tuning for Divide-and-conquer Kernel Ridge Regression with Massive Data

We propose a first data-driven tuning procedure for divide-and-conquer kernel ridge regression (Zhang et al., 2015). While the proposed criterion is computationally scalable for massive data sets, it is also shown to be asymptotically optimal under mild conditions. The effectiveness of our method is illustrated by extensive simulations and an application to Million Song Dataset.

[1]  Lei Shi,et al.  Learning Theory of Distributed Regression with Bias Corrected Regularization Kernel Network , 2017, J. Mach. Learn. Res..

[2]  G. Blanchard,et al.  Parallelizing Spectral Algorithms for Kernel Learning , 2016, 1610.07487.

[3]  Ding-Xuan Zhou,et al.  Distributed Learning with Regularized Least Squares , 2016, J. Mach. Learn. Res..

[4]  Han Liu,et al.  Nonparametric Heterogeneity Testing For Massive Data , 2016, 1601.06212.

[5]  Guang Cheng,et al.  Computational Limits of A Distributed Algorithm for Smoothing Spline , 2015, J. Mach. Learn. Res..

[6]  Han Liu,et al.  A PARTIALLY LINEAR FRAMEWORK FOR MASSIVE HETEROGENEOUS DATA. , 2014, Annals of statistics.

[7]  Minge Xie,et al.  A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data , 2014 .

[8]  Martin J. Wainwright,et al.  Divide and conquer kernel ridge regression: a distributed algorithm with minimax optimal rates , 2013, J. Mach. Learn. Res..

[9]  Jianhua Z. Huang,et al.  Asymptotic optimality and efficient computation of the leave-subject-out cross-validation , 2012, 1302.4607.

[10]  Chong Gu,et al.  Optimal smoothing in nonparametric mixed-effect models , 2005, math/0507428.

[11]  S. Wood.  Stable and Efficient Multiple Smoothing Parameter Estimation for Generalized Additive Models , 2004 .

[12]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[13]  S. R. Jammalamadaka,et al.  Empirical Processes in M-Estimation , 2001 .

[14]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[15]  Ker-Chau Li,et al.  Asymptotic optimality of CL and generalized cross-validation in ridge regression with application to spline smoothing , 1986 .

[16]  M. Rosenblatt,et al.  Smoothing Splines: Regression, Derivatives and Deconvolution , 1983 .

[17]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[18]  Thierry Bertin-Mahieux,et al.  The Million Song Dataset , 2011, ISMIR.

[19]  N. Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[20]  G. Wahba,et al.  A GENERALIZED APPROXIMATE CROSS VALIDATION FOR SMOOTHING SPLINES WITH NON-GAUSSIAN DATA , 1996 .

[21]  D. Pollard.  Uniform ratio limit theorems for empirical processes , 1995 .

[22]  Francis R. Bach.  Sharp analysis of low-rank kernel matrix approximations , 2012, COLT.