Nested model averaging on solution path for high‐dimensional linear regression

We study the nested model averaging method on the solution path for a high-dimensional linear regression problem. In particular, we propose to combine model averaging with regularized estimators (e.g., lasso and SLOPE) on the solution path for high-dimensional linear regression. In simulation studies, we first conduct a systematic investigation on the impact of predictor ordering on the behavior of nested model averaging, then show that nested model averaging with lasso and SLOPE compares favorably with other competing methods, including the infeasible lasso and SLOPE with the tuning parameter optimally selected. A real data analysis on predicting the per capita violent crime in the United States shows an outstanding performance of the nested model averaging with lasso.

[1]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[2]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[3]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[4]  Nils Lid Hjort,et al.  Model Selection and Model Averaging , 2001 .

[5]  Yang Feng,et al.  On the sparsity of Mallows model averaging estimator , 2020 .

[6]  Hua Liang,et al.  Optimal Model Averaging Estimation for Generalized Linear Models and Generalized Linear Mixed-Effects Models , 2016 .

[7]  Xun Lu,et al.  Jackknife Model Averaging for Quantile Regressions , 2014 .

[8]  Mark F.J. Steel,et al.  Model Averaging and Its Use in Economics , 2017, Journal of Economic Literature.

[9]  Yang Feng,et al.  Model Selection for High-Dimensional Quadratic Regression via Regularization , 2014, 1501.00049.

[10]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[11]  U. C. Bureau,et al.  Census of Population and Housing , 1993 .

[12]  A. Belloni,et al.  Least Squares After Model Selection in High-Dimensional Sparse Models , 2009, 1001.0188.

[13]  Qingfeng Liu,et al.  Model averaging estimation for conditional volatility models with an application to stock market volatility forecast , 2020 .

[14]  Sara van de Geer,et al.  Statistics for High-Dimensional Data: Methods, Theory and Applications , 2011 .

[15]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[16]  N. Hjort,et al.  Comprar Model Selection and Model Averaging | Gerda Claeskens | 9780521852258 | Cambridge University Press , 2008 .

[17]  B. Hansen.  Least Squares Model Averaging , 2007 .

[18]  Helmut Moritz,et al.  A generalized least-squares model , 1970 .

[19]  Enrique Moral-Benito,et al.  Model Averaging in Economics: An Overview , 2015 .

[20]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[21]  Guohua Zou,et al.  Least squares model averaging by Mallows criterion , 2010 .

[22]  Jiahua Chen,et al.  Extended Bayesian information criteria for model selection with large model spaces , 2008 .

[23]  Bruce E. Hansen,et al.  Model averaging, asymptotic risk, and regressor groups , 2014 .

[24]  Emmanuel J. Candès,et al.  SLOPE is Adaptive to Unknown Sparsity and Asymptotically Minimax , 2015, ArXiv.

[25]  Ryo Okui,et al.  Generalized Least Squares Model Averaging , 2014 .

[26]  Ryo Okui,et al.  Constructing Optimal Instruments by First-Stage Prediction Averaging , 2010 .

[27]  Emmanuel J. Candès,et al.  False Discoveries Occur Early on the Lasso Path , 2015, ArXiv.

[28]  Weijie J. Su,et al.  SLOPE-ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION. , 2014, The annals of applied statistics.

[29]  Jianqing Fan,et al.  A Selective Overview of Variable Selection in High Dimensional Feature Space. , 2009, Statistica Sinica.

[30]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[31]  Xu Cheng,et al.  Forecasting with Factor-Augmented Regression: A Frequentist Model Averaging Approach , 2012 .

[32]  Ryo Okui,et al.  Heteroscedasticity�?Robust C Model Averaging , 2012 .

[33]  Yingying Fan,et al.  Tuning parameter selection in high dimensional penalized likelihood , 2013, 1605.03321.

[34]  Alok Baveja,et al.  Computing , Artificial Intelligence and Information Technology A data-driven software tool for enabling cooperative information sharing among police departments , 2002 .

[35]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[36]  Yi Yu,et al.  The restricted consistency property of leave-nv-out cross-validation for high-dimensional variable selection , 2013, Statistica Sinica.

[37]  Sara van de Geer,et al.  Statistics for High-Dimensional Data , 2011 .