Smoothing spline Gaussian regression: more scalable computation via efficient approximation

Summary.  Smoothing splines via the penalized least squares method provide versatile and effective nonparametric models for regression with Gaussian responses. The computation of smoothing splines is generally of the order O(n3), n being the sample size, which severely limits its practical applicability. We study more scalable computation of smoothing spline regression via certain low dimensional approximations that are asymptotically as efficient. A simple algorithm is presented and the Bayes model that is associated with the approximations is derived, with the latter guiding the porting of Bayesian confidence intervals. The practical choice of the dimension of the approximating space is determined through simulation studies, and empirical comparisons of the approximations with the exact solution are presented. Also evaluated is a simple modification of the generalized cross‐validation method for smoothing parameter selection, which to a large extent fixes the occasional undersmoothing problem that is suffered by generalized cross‐validation.

[1]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[2]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[3]  G. Wahba,et al.  SPLINE FUNCTIONS AND STOCHASTIC PROCESSES. , 1969 .

[4]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[5]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[6]  G. Wahba.  Smoothing noisy data with spline functions , 1975 .

[7]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[8]  G. Wahba.  Improper Priors, Spline Smoothing and the Problem of Guarding Against Model Errors in Regression , 1978 .

[9]  Steven A. Orszag,et al.  CBMS-NSF REGIONAL CONFERENCE SERIES IN APPLIED MATHEMATICS , 1978 .

[10]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[11]  G. Wahba.  Spline Interpolation and Smoothing on the Sphere , 1981 .

[12]  G. Wahba.  Bayesian "Confidence Intervals" for the Cross-validated Smoothing Spline , 1983 .

[13]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[14]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation. , 1985 .

[15]  G. Wahba.  A Comparison of GCV and GML for Choosing the Smoothing Parameter in the Generalized Spline Smoothing Problem , 1985 .

[16]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation: Rejoinder , 1985 .

[17]  Ker-Chau Li,et al.  Asymptotic optimality of CL and generalized cross-validation in ridge regression with application to spline smoothing , 1986 .

[18]  Douglas Nychka,et al.  Bayesian Confidence Intervals for Smoothing Splines , 1988 .

[19]  G. Wahba,et al.  The computation of generalized cross-validation functions through householder tridiagonalization with applications to the fitting of interaction spline models , 1989 .

[20]  R. Tibshirani,et al.  Linear Smoothers and Additive Models , 1989 .

[21]  G. Wahba.  Spline models for observational data , 1990 .

[22]  Chong Gu,et al.  Minimizing GCV/GML Scores with Multiple Smoothing Parameters via the Newton Method , 1991, SIAM J. Sci. Comput..

[23]  G. Wahba,et al.  Smoothing Spline ANOVA with Component-Wise Bayesian “Confidence Intervals” , 1993 .

[24]  G. Wahba,et al.  Semiparametric Analysis of Variance with Tensor Product Thin Plate Splines , 1993 .

[25]  B. Silverman,et al.  Nonparametric regression and generalized linear models , 1994 .

[26]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[27]  G. Wahba,et al.  Hybrid Adaptive Splines , 1997 .

[28]  S. Wood.  Modelling and smoothing parameter estimation with multiple quadratic penalties , 2000 .

[29]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[30]  Chong Gu,et al.  Penalized likelihood regression: General formulation and efficient approximation , 2002 .

[31]  Chong Gu,et al.  PENALIZED LIKELIHOOD DENSITY ESTIMATION: DIRECT CROSS-VALIDATION AND SCALABLE APPROXIMATION , 2003 .