On algorithms for ordinary least squares regression spline fitting: A comparative study

Regression spline smoothing is a popular approach for conducting nonparametric regression. An important issue associated with it is the choice of a "theoretically best" set of knots. Different statistical model selection methods, such as Akaike's information criterion and generalized cross-validation, have been applied to derive different "theoretically best" sets of knots. Typically these best knot sets are defined implicitly as the optimizers of some objective functions. Hence another equally important issue concerning regression spline smoothing is how to optimize such objective functions. In this article different numerical algorithms that are designed for carrying out such optimization problems are compared by means of a simulation study. Both the univariate and bivariate smoothing settings will be considered. Based on the simulation results, recommendations for choosing a suitable optimization algorithm under various settings will be provided.

[1]  C. A. Murthy,et al.  Fitting Optimal Piecewise Linear Functions Using Genetic Algorithms , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  C. J. Stone,et al.  Polychotomous Regression , 1995 .

[3]  Thomas C.M. Lee,et al.  Regression spline smoothing using the minimum description length principle , 2000 .

[4]  Matt P. Wand,et al.  A Comparison of Regression Spline Smoothing Procedures , 2000, Comput. Stat..

[5]  David G. T. Denison,et al.  Bayesian MARS , 1998, Stat. Comput..

[6]  Mary J. Lindstrom,et al.  Penalized Estimation of Free-Knot Splines , 1999 .

[7]  Trevor Hastie,et al.  Polynomial splines and their tensor products in extended linear modeling. Discussion and rejoinder , 1997 .

[8]  J. Friedman,et al.  FLEXIBLE PARSIMONIOUS SMOOTHING AND ADDITIVE MODELING , 1989 .

[9]  R. Kohn,et al.  Nonparametric regression using Bayesian variable selection , 1996 .

[10]  R. Kass,et al.  Bayesian curve-fitting with free-knot splines , 2001 .

[11]  Trevor Hastie,et al.  [Flexible Parsimonious Smoothing and Additive Modeling]: Discussion , 1989 .

[12]  Young K. Truong,et al.  Polynomial splines and their tensor products in extended linear modeling: 1994 Wald memorial lecture , 1997 .

[13]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[14]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[15]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[16]  B. Silverman,et al.  Nonparametric regression and generalized linear models , 1994 .

[17]  Young K. Truong,et al.  Polynomial splines and their tensor products in extended linearmodeling , 1997 .

[18]  Ja-Yong Koo,et al.  Spline Estimation of Discontinuous Regression Functions , 1997 .

[19]  David Ruppert,et al.  Theory & Methods: Spatially‐adaptive Penalties for Spline Fitting , 2000 .

[20]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[21]  Jennifer Pittman,et al.  Adaptive Splines and Genetic Algorithms , 2000 .

[22]  Thomas C.M. Lee,et al.  AUTOMATIC SMOOTHING FOR DISCONTINUOUS REGRESSION FUNCTIONS , 2002 .

[23]  Adrian F. M. Smith,et al.  Automatic Bayesian curve fitting , 1998 .

[24]  S. Sardy,et al.  Triogram Models , 1996 .

[25]  Jerome H. Friedman.  Multivariate adaptive regression splines (with discussion) , 1991 .

[26]  A. F. M. Smith,et al.  Automatic Bayesian curve ® tting , 1998 .

[27]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[28]  David B. Fogel.  What is evolutionary computation , 1995 .

[29]  Paul H. C. Eilers,et al.  Flexible smoothing with B-splines and penalties , 1996 .