COMPONENT SELECTION AND SMOOTHING FOR NONPARAMETRIC REGRESSION IN EXPONENTIAL FAMILIES

We propose a new penalized likelihood method for model selection and nonparametric regression in exponential families. In the framework of smoothing spline ANOVA, our method employs a regularization with the penalty functional being the sum of the reproducing kernel Hilbert space norms of functional com- ponents in the ANOVA decomposition. It generalizes the LASSO in the linear regression to the nonparametric context, and conducts component selection and smoothing simultaneously. Continuous and categorical variables are treated in a unied fashion. We discuss the connection of the method to the traditional smooth- ing spline penalized likelihood estimation. We show that an equivalent formulation of the method leads naturally to an iterative algorithm. Simulations and examples are used to demonstrate the performances of the method.

[1]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[2]  R. Tapia,et al.  Nonparametric Probability Density Estimation , 1978 .

[3]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[4]  Douglas M. Hawkins,et al.  Discussion of ‘Flexible Parsimonious Smoothing and Additive Modeling’ , 1989 .

[5]  J. Friedman,et al.  FLEXIBLE PARSIMONIOUS SMOOTHING AND ADDITIVE MODELING , 1989 .

[6]  G. Wahba.  Spline models for observational data , 1990 .

[7]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[8]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[9]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[10]  Chong Gu.  Diagnostics for Nonparametric Regression Models with Additive Terms , 1992 .

[11]  G. Wahba,et al.  Smoothing spline ANOVA for exponential families, with application to the Wisconsin Epidemiological Study of Diabetic Retinopathy : the 1994 Neyman Memorial Lecture , 1995 .

[12]  L. Breiman.  Better subset regression using the nonnegative garrote , 1995 .

[13]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[14]  Chaogang Xiang.  Approximate Smoothing Spline Methods for Large DataSets in the Binary , 1997 .

[15]  Jianqing Fan,et al.  Variable Selection via Penalized Likelihood , 1999 .

[16]  David Ruppert,et al.  Theory & Methods: Spatially‐adaptive Penalties for Spline Fitting , 2000 .

[17]  Chong Gu.  Smoothing Spline Anova Models , 2002 .

[18]  Chong Gu,et al.  Penalized likelihood regression: General formulation and efficient approximation , 2002 .

[19]  Hao Helen Zhang,et al.  Component selection and smoothing in smoothing spline analysis of variance models -- COSSO , 2003 .

[20]  Robert Kohn,et al.  Bayesian Variable Selection and Model Averaging in High-Dimensional Multinomial Nonparametric Regression , 2003 .

[21]  Michael I. Jordan,et al.  Computing regularization paths for learning multiple kernels , 2004, NIPS.

[22]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[23]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[24]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[25]  Meta M. Voelker,et al.  Variable Selection and Model Building via Likelihood Basis Pursuit , 2004 .

[26]  Hao Helen Zhang,et al.  Component selection and smoothing in multivariate nonparametric regression , 2006, math/0702659.