Linear Smoothers and Additive Models

We study linear smoothers and their use in building nonparametric regression models. In the first part of this paper we examine certain aspects of linear smoothers for scatterplots; examples of these are the running-mean and running-line, kernel and cubic spline smoothers. The eigenvalue and singular value decompositions of the corresponding smoother matrix are used to describe qualitatively a smoother, and several other topics such as the number of degrees of freedom of a smoother are discussed. In the second part of the paper we describe how linear smoothers can be used to estimate the additive model, a powerful nonparametric regression model, using the "backfitting algorithm." We show that backfitting is the Gauss-Seidel iterative method for solving a set of normal equations associated with the additive model. We provide conditions for consistency and nondegeneracy and prove convergence for the backfitting and related algorithms for a class of smoothers that includes cubic spline smoothers.

[1]  Edmund Taylor Whittaker.  On a New Method of Graduation , 1922, Proceedings of the Edinburgh Mathematical Society.

[2]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[3]  Alston S. Householder,et al.  The Theory of Matrices in Numerical Analysis , 1964 .

[4]  H. Keller.  On the Solution of Singular and Semidefinite Linear Systems by Iteration , 1965 .

[5]  Tosio Kato.  Perturbation theory for linear operators , 1966 .

[6]  C. Reinsch.  Smoothing by spline functions , 1967 .

[7]  M. Priestley,et al.  Non‐Parametric Function Fitting , 1972 .

[8]  G. Wahba.  Smoothing noisy data with spline functions , 1975 .

[9]  C. Reinsch,et al.  Oscillation matrices with spline smoothing , 1975 .

[10]  C. J. Stone,et al.  Consistent Nonparametric Regression , 1977 .

[11]  John W. Tukey,et al.  Exploratory Data Analysis. , 1979 .

[12]  Frederick Mosteller,et al.  Data Analysis and Regression , 1978 .

[13]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[14]  F. Utreras.  Cross-validation techniques for smoothing spline functions in one or two dimensions , 1979 .

[15]  W. Cleveland.  Robust Locally Weighted Regression and Smoothing Scatterplots , 1979 .

[16]  H. Müller,et al.  Kernel estimation of regression functions , 1979 .

[17]  C. L. Mallows,et al.  Some Theory of Nonlinear Smoothers , 1980 .

[18]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[19]  Jerome H. Friedman,et al.  Smoothing of Scatterplots , 1982 .

[20]  Jan de Leeuw,et al.  Non-linear canonical correlation , 1983 .

[21]  G. Wahba.  Bayesian "Confidence Intervals" for the Cross-validated Smoothing Spline , 1983 .

[22]  W. Cleveland.  Seasonal and calendar adjustment , 1983 .

[23]  M. Rosenblatt,et al.  Smoothing Splines: Regression, Derivatives and Deconvolution , 1983 .

[24]  J. Friedman,et al.  Multidimensional Additive Spline Approximation , 1983 .

[25]  D. Cox.  Asymptotics for $M$-Type Smoothing Splines , 1983 .

[26]  Gene H. Golub,et al.  Matrix computations , 1983 .

[27]  Andrew F. Siegel,et al.  Low Median and Least Absolute Residual Analysis of Two-Way Tables , 1983 .

[28]  J. Kemperman.  LEAST ABSOLUTE VALUE AND MEDIAN POLISH , 1984 .

[29]  P. Green.  Iteratively reweighted least squares for maximum likelihood estimation , 1984 .

[30]  R. L. Eubank,et al.  The hat matrix for smoothing splines , 1984 .

[31]  B. Silverman,et al.  Spline Smoothing: The Equivalent Variable Kernel Method , 1984 .

[32]  C. J. Stone,et al.  Additive Regression and Other Nonparametric Models , 1985 .

[33]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation. , 1985 .

[34]  B. Yandell,et al.  Semi-Parametric Generalized Linear Models. , 1985 .

[35]  A. Seheult,et al.  Analysis of Field Experiments by Least Squares Smoothing , 1985 .

[36]  Will Light,et al.  Approximation Theory in Tensor Product Spaces , 1985 .

[37]  G. Wahba.  Partial and interaction spline models for the semiparametric estimation of functions of several variables , 1986 .

[38]  B. Yandell,et al.  Automatic Smoothing of Regression Functions in Generalized Linear Models , 1986 .

[39]  F. O’Sullivan.  A Statistical Perspective on Ill-posed Inverse Problems , 1986 .

[40]  [A Statistical Perspective on Ill-Posed Inverse Problems]: Comment , 1986 .

[41]  Colin L. Mallows,et al.  Augmented partial residuals , 1986 .

[42]  R. Tibshirani,et al.  Generalized Additive Models, Cubic Splines and Penalized Likelihood. , 1987 .

[43]  Wolfgang Härdle.  Resistant Smoothing Using the Fast Fourier Transform , 1987 .

[44]  R. Tibshirani,et al.  Generalized Additive Models: Some Applications , 1987 .

[45]  P. Green.  Penalized Likelihood for General Semi-Parametric Regression Models. , 1987 .

[46]  R. Tibshirani,et al.  Local Likelihood Estimation , 1987 .

[47]  Trevor Hastie,et al.  [Monotone Regression Splines in Action]: Comment , 1988 .

[48]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[49]  W. Cleveland,et al.  Regression by local fitting: Methods, properties, and computational algorithms , 1988 .

[50]  W. Cleveland,et al.  Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting , 1988 .

[51]  J. Ramsay.  Monotone Regression Splines in Action , 1988 .

[52]  Trevor Hastie,et al.  [Flexible Parsimonious Smoothing and Additive Modeling]: Discussion , 1989 .

[53]  J. Friedman,et al.  FLEXIBLE PARSIMONIOUS SMOOTHING AND ADDITIVE MODELING , 1989 .

[54]  A. Buja.  Remarks on Functional Canonical Variates, Alternating Least Squares Methods and Ace , 1990 .

[55]  Prabir Burman,et al.  Estimation of generalized additive models , 1990 .