Adaptive Model Selection

Most model selection procedures use a fixed penalty penalizing an increase in the size of a model. These nonadaptive selection procedures perform well only in one type of situation. For instance, Bayesian information criterion (BIC) with a large penalty performs well for “small” models and poorly for “large” models, and Akaike's information criterion (AIC) does just the opposite. This article proposes an adaptive model selection procedure that uses a data-adaptive complexity penalty based on a concept of generalized degrees of freedom. The proposed procedure, combining the benefit of a class of nonadaptive procedures, approximates the best performance of this class of procedures across a variety of different situations. This class includes many well-known procedures, such as AIC, BIC, Mallows's Cp, and risk inflation criterion (RIC). The proposed procedure is applied to wavelet thresholding in nonparametric regression and variable selection in least squares regression. Simulation results and an asymptotic analysis support the effectiveness of the proposed procedure.

[1]  C. L. Mallows Some comments on C_p , 1973 .

[2]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[3]  C. Stein Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[4]  J. Rice Bandwidth Choice for Nonparametric Regression , 1984 .

[5]  K. Alexander,et al.  Probability Inequalities for Empirical Processes and a Law of the Iterated Logarithm , 1984 .

[6]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[7]  Y. Meyer,et al.  Remarques sur l'analyse de Fourier à fenêtre , 1991 .

[8]  A. Atkinson Subset Selection in Regression , 1992 .

[9]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[10]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[11]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[12]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[13]  Y. Benjamini,et al.  Adaptive thresholding of wavelet coefficients , 1996 .

[14]  Hong-Ye Gao,et al.  Applied wavelet analysis with S-plus , 1996 .

[15]  Dean Phillips Foster,et al.  An Information Theoretic Comparison of Model Selection Criteria , 1997 .

[16]  Jianming Ye On Measuring and Correcting the Effects of Data Mining and Model Selection , 1998 .

[17]  Michael H. Neumann,et al.  Exact Risk Analysis of Wavelet Regression , 1998 .

[18]  R. Tibshirani,et al.  The Covariance Inflation Criterion for Adaptive Model Selection , 1999 .

[19]  Jonnagadda S Rao,et al.  Bootstrap choice of cost complexity for better subset selection , 1999 .

[20]  Yuehua Wu,et al.  Model selection with data-oriented penalty , 1999 .

[21]  Noel Cressie,et al.  Deterministic/Stochastic Wavelet Decomposition for Recovery of Signal From Noisy Data , 2000, Technometrics.

[22]  Jun Shao,et al.  The gic for model selection : a hypothesis testing approach , 2000 .

[23]  Faming Liang,et al.  Automatic Bayesian model averaging for linear regression and applications in Bayesian curve fitting , 2001 .