Low Rank Approximation for Smoothing Spline via Eigensystem Truncation

Smoothing splines provide a powerful and flexible means for nonparametric estimation and inference. With a cubic time complexity, fitting smoothing spline models to large data is computationally prohibitive. In this paper, we use the theoretical optimal eigenspace to derive a low rank approximation of the smoothing spline estimates. We develop a method to approximate the eigensystem when it is unknown and derive error bounds for the approximate estimates. The proposed methods are easy to implement with existing software. Extensive simulations show that the new methods are accurate, fast, and compares favorably against existing methods.

[1]  D. Bates,et al.  Newton-Raphson and EM Algorithms for Linear Mixed-Effects Models for Repeated-Measures Data , 1988 .

[2]  Ameet Talwalkar,et al.  On the Impact of Kernel Approximation on Learning Accuracy , 2010, AISTATS.

[3]  Yuedong Wang,et al.  Mixed-Effects Smoothing Spline ANOVA , 1998 .

[4]  Michael W. Mahoney,et al.  Fast Randomized Kernel Ridge Regression with Statistical Guarantees , 2015, NIPS.

[5]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[6]  R. Taylor,et al.  The Numerical Treatment of Integral Equations , 1978 .

[7]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[8]  G. Wahba.  Spline models for observational data , 1990 .

[9]  Martin J. Wainwright,et al.  Randomized sketches for kernels: Fast and optimal non-parametric regression , 2015, ArXiv.

[10]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[11]  Danqing Xu,et al.  Divide and Recombine Approaches for Fitting Smoothing Spline Models with Large Datasets , 2018, Journal of Computational and Graphical Statistics.

[12]  L. Delves,et al.  Computational methods for integral equations: Frontmatter , 1985 .

[13]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[14]  Jianhua Z. Huang,et al.  Efficient computation of smoothing splines via adaptive basis sampling , 2015 .

[15]  Yuedong Wang,et al.  Smoothing Splines: Methods and Applications , 2011 .

[16]  Robert Schaback,et al.  Approximation of eigenfunctions in kernel-based spaces , 2014, Adv. Comput. Math..

[17]  S. Wood.  Thin plate regression splines , 2003 .

[18]  Chong Gu,et al.  Smoothing spline Gaussian regression: more scalable computation via efficient approximation , 2004 .

[19]  M. Wainwright,et al.  Sampled forms of functional PCA in reproducing kernel Hilbert spaces , 2011, 1109.3336.

[20]  G. Wahba,et al.  Design Problems for Optimal Surface Interpolation. , 1979 .

[21]  Charles A. Micchelli,et al.  SPLINE SPACES ARE OPTIMAL FOR L 2 n-WIDTH , 1978 .

[22]  Mark Girolami,et al.  Orthogonal Series Density Estimation and the Kernel Eigenvalue Problem , 2002, Neural Computation.

[23]  Francis R. Bach,et al.  Sharp analysis of low-rank kernel matrix approximations , 2012, COLT.

[24]  Yuedong Wang,et al.  ASSIST: A Suite of S functions Implementing Spline smoothing Techniques , 2014 .