Smooth predictive model fitting in regression

We propose a smooth hypothesis-testing type method for model fitting in regression and develop its theoretical properties in a moderately high-dimensional setting. We derive the asymptotic behavior of the L 2 prediction risk and the optimal choice of the threshold-determining parameter under the Gaussian regression framework with orthogonal covariates. When the covariates are not orthogonal, this method can be used in conjunction with some reasonable procedures for orthogonalizing the covariates, such as a stepwise regression coupled with the Gram-Schmidt procedure. Simulation results show that our proposed method often outperforms its competitors. Asymptotic approximation of the mean square error of the proposed estimator is obtained and it is shown that the optimal rate of convergence depends on the behavior of the smoothing function at zero.

[1]  Jiahua Chen,et al.  Extended Bayesian information criteria for model selection with large model spaces , 2008 .

[2]  H. Akaike.  Statistical predictor identification , 1970 .

[3]  David Alan Grier,et al.  Test-Based Variable Selection via Cross-Validation , 1992 .

[4]  Tao Wang,et al.  Consistent tuning parameter selection in high dimensional sparse linear regression , 2011, J. Multivar. Anal..

[5]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[6]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[7]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[8]  Rudolf Beran,et al.  React Scatterplot Smoothers: Superefficiency through Basis Economy , 2000 .

[9]  Prabir Burman,et al.  MODEL FITTING VIA TESTING , 1996 .

[10]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[11]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[12]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[13]  Christopher M. Bishop,et al.  Classification and regression , 1997 .

[14]  M. Stone.  Continuum regression: Cross-validated sequentially constructed prediction embracing ordinary least s , 1990 .

[15]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[16]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[17]  Prabir Burman,et al.  Nonparametric Estimation of a Regression Function , 1989 .

[18]  Ping Zhang.  Model Selection Via Multifold Cross Validation , 1993 .

[19]  P. Burman.  A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods , 1989 .

[20]  R. Shibata.  An optimal selection of regression variables , 1981 .

[21]  Ker-Chau Li,et al.  Asymptotic Optimality for $C_p, C_L$, Cross-Validation and Generalized Cross-Validation: Discrete Index Set , 1987 .

[22]  C. Mallows.  More comments on C p , 1995 .

[23]  Chenlei Leng,et al.  Shrinkage tuning parameter selection with a diverging number of parameters , 2008 .

[24]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[25]  R. Rao,et al.  Normal Approximation and Asymptotic Expansions , 1976 .

[26]  Sara van de Geer,et al.  Statistics for High-Dimensional Data , 2011 .

[27]  C. L. Mallows.  Some comments on C_p , 1973 .

[28]  I. Johnstone,et al.  Wavelet Shrinkage: Asymptopia? , 1995 .

[29]  Yingying Fan,et al.  Tuning parameter selection in high dimensional penalized likelihood , 2013, 1605.03321.