Tuning parameter selection in sparse regression modeling

In sparse regression modeling via regularization such as the lasso, it is important to select appropriate values of tuning parameters including regularization parameters. The choice of tuning parameters can be viewed as a model selection and evaluation problem. Mallows' C"p type criteria may be used as a tuning parameter selection tool in lasso type regularization methods, for which the concept of degrees of freedom plays a key role. In the present paper, we propose an efficient algorithm that computes the degrees of freedom by extending the generalized path seeking algorithm. Our procedure allows us to construct model selection criteria for evaluating models estimated by regularization with a wide variety of convex and nonconvex penalties. The proposed methodology is investigated through the analysis of real data and Monte Carlo simulations. Numerical results show that C"p criterion based on our algorithm performs well in various situations.

[1]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[2]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[3]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[4]  Chenlei Leng,et al.  Shrinkage tuning parameter selection with a diverging number of parameters , 2008 .

[5]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[6]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[7]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[8]  R. Tibshirani,et al.  The solution path of the generalized lasso , 2010, 1005.1971.

[9]  N. Sugiura.  Further analysts of the data by akaike' s information criterion and the finite corrections , 1978 .

[10]  P. Zhao,et al.  The composite absolute penalties family for grouped and hierarchical variable selection , 2009, 0909.0411.

[11]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[12]  Hideo Hirose,et al.  NNRMLR: A Combined Method of Nearest Neighbor Regression and Multiple Linear Regression , 2012, 2012 IIAI International Conference on Advanced Applied Informatics.

[13]  Bertrand Michel,et al.  Slope heuristics: overview and implementation , 2011, Statistics and Computing.

[14]  C. L. Mallows.  Some comments on C_p , 1973 .

[15]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[16]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[17]  Clifford M. Hurvich,et al.  Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion , 1998 .

[18]  Xiaotong Shen,et al.  Adaptive Model Selection and Assessment for Exponential Family Distributions , 2004, Technometrics.

[19]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[20]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[21]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[22]  Yixin Fang,et al.  A note on the generalized degrees of freedom under the L1 loss function , 2011 .

[23]  R. Tibshirani,et al.  On the “degrees of freedom” of the lasso , 2007, 0712.0881.

[24]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[25]  G. Kitagawa,et al.  Information Criteria and Statistical Modeling , 2007 .

[26]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[27]  Jianming Ye.  On Measuring and Correcting the Effects of Data Mining and Model Selection , 1998 .

[28]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[29]  Wenjiang J. Fu.  Penalized Regressions: The Bridge versus the Lasso , 1998 .

[30]  Kengo Kato,et al.  On the degrees of freedom in shrinkage estimation , 2009, J. Multivar. Anal..

[31]  J. Friedman.  Fast sparse regression and classification , 2012 .

[32]  T. Hastie,et al.  SparseNet: Coordinate Descent With Nonconvex Penalties , 2011, Journal of the American Statistical Association.

[33]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[34]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[35]  Xiaotong Shen,et al.  Adaptive Model Selection , 2002 .

[36]  Runze Li,et al.  Tuning parameter selectors for the smoothly clipped absolute deviation method. , 2007, Biometrika.

[37]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[38]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[39]  B. Efron.  The Estimation of Prediction Error , 2004 .

[40]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .