Model Selection for High-Dimensional Quadratic Regression via Regularization

ABSTRACT Quadratic regression (QR) models naturally extend linear models by considering interaction effects between the covariates. To conduct model selection in QR, it is important to maintain the hierarchical model structure between main effects and interaction effects. Existing regularization methods generally achieve this goal by solving complex optimization problems, which usually demands high computational cost and hence are not feasible for high-dimensional data. This article focuses on scalable regularization methods for model selection in high-dimensional QR. We first consider two-stage regularization methods and establish theoretical properties of the two-stage LASSO. Then, a new regularization method, called regularization algorithm under marginality principle (RAMP), is proposed to compute a hierarchy-preserving regularization solution path efficiently. Both methods are further extended to solve generalized QR models. Numerical results are also shown to demonstrate performance of the methods. Supplementary materials for this article are available online.

[1]  P. Zhao,et al.  The composite absolute penalties family for grouped and hierarchical variable selection , 2009, 0909.0411.

[2]  Bo Jiang,et al.  Variable selection for general index models via sliced inverse regression , 2013, 1304.4056.

[3]  Hansheng Wang.  Forward Regression for Ultra-High Dimensional Variable Screening , 2009 .

[4]  Hua Zhou,et al.  A Generic Path Algorithm for Regularized Statistical Estimation , 2012, Journal of the American Statistical Association.

[5]  Jianqing Fan,et al.  Sure independence screening for ultrahigh dimensional feature space , 2006, math/0612857.

[6]  Changbao Wu,et al.  Analysis of Designed Experiments with Complex Aliasing , 1992 .

[7]  Trevor J. Hastie,et al.  Genome-wide association analysis by lasso penalized logistic regression , 2009, Bioinform..

[8]  J. Nelder.  A Reformulation of Linear Models , 1977 .

[9]  Yingying Fan,et al.  Tuning parameter selection in high dimensional penalized likelihood , 2013, 1605.03321.

[10]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[11]  Ning Hao,et al.  Interaction Screening for Ultra-High Dimensional Data. , 2014, Journal of the American Statistical Association.

[12]  Ning Hao,et al.  A Note on High-Dimensional Linear Regression With Interactions , 2014, 1412.7138.

[13]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[14]  Ji Zhu,et al.  Variable Selection With the Strong Heredity Constraint and Its Oracle Property , 2010 .

[15]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[16]  Jiahua Chen,et al.  Extended Bayesian information criteria for model selection with large model spaces , 2008 .

[17]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[18]  Jianqing Fan,et al.  Nonconcave Penalized Likelihood With NP-Dimensionality , 2009, IEEE Transactions on Information Theory.

[19]  H. Chipman,et al.  A Bayesian variable-selection approach for analyzing designed experiments with complex aliasing , 1997 .

[20]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[21]  Yang Feng,et al.  Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Additive Models , 2009, Journal of the American Statistical Association.

[22]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[23]  H. Zou,et al.  Structured variable selection and estimation , 2009, 1011.0610.

[24]  K. Lange,et al.  Coordinate descent algorithms for lasso penalized regression , 2008, 0803.3876.

[25]  W Y Zhang,et al.  Discussion on `Sure independence screening for ultra-high dimensional feature space' by Fan, J and Lv, J. , 2008 .

[26]  Yingying Fan,et al.  Interaction pursuit in high-dimensional multi-response regression via distance correlation , 2016, 1605.03315.

[27]  Yichao Wu,et al.  An ordinary differential equation-based solution path algorithm , 2011, Journal of nonparametric statistics.

[28]  Yang Feng,et al.  APPLE: approximate path for penalized likelihood estimators , 2012, Stat. Comput..

[29]  J. Peixoto.  Hierarchical Variable Selection in Polynomial Regression Models , 1987 .

[30]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[31]  Jialiang Li,et al.  Nonparametric independence screening and structure identification for ultra-high dimensional longitudinal data , 2013, 1308.3942.

[32]  Yingying Fan,et al.  Interaction Pursuit with Feature Screening and Selection , 2016, 1605.08933.

[33]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[34]  Mee Young Park,et al.  L1‐regularization path algorithm for generalized linear models , 2007 .

[35]  Yingying Fan,et al.  Asymptotic Equivalence of Regularization Methods in Thresholded Parameter Space , 2013, 1605.03310.

[36]  Ning Hao,et al.  Interaction Screening for Ultrahigh-Dimensional Data , 2014, Journal of the American Statistical Association.

[37]  R. Tibshirani,et al.  A LASSO FOR HIERARCHICAL INTERACTIONS. , 2012, Annals of statistics.

[38]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[39]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[40]  Yingying Fan,et al.  INNOVATED INTERACTION SCREENING FOR HIGH-DIMENSIONAL NONLINEAR CLASSIFICATION , 2015 .

[41]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.