Factor-Adjusted Regularized Model Selection

This paper studies model selection consistency for high dimensional sparse regression when data exhibits both cross-sectional and serial dependency. Most commonly-used model selection methods fail to consistently recover the true model when the covariates are highly correlated. Motivated by econometric and financial studies, we consider the case where covariate dependence can be reduced through the factor model, and propose a consistency strategy named Factor-Adjusted Regularized Model Selection (FarmSelect). By learning the latent factors and idiosyncratic components and using both of them as predictors, FarmSelect transforms the problem from model selection with highly correlated covariates to that with weakly correlated ones via lifting. Model selection consistency, as well as optimal rates of convergence, are obtained under mild conditions. Numerical studies demonstrate the nice finite sample performance in terms of both model selection and out-of-sample prediction. Moreover, our method is flexible in the sense that it pays no price for weakly correlated and uncorrelated cases. Our method is applicable to a wide range of high dimensional sparse regression problems. An R-package FarmSelect is also provided for implementation.

[1]  Jianqing Fan,et al.  Robust high dimensional factor models with applications to statistical machine learning. , 2018, Statistical science : a review journal of the Institute of Mathematical Statistics.

[2]  Jianqing Fan,et al.  Asymptotics of empirical eigenstructure for high dimensional spiked covariance. , 2017, Annals of statistics.

[3]  Hongtu Zhu,et al.  The Statistics and Mathematics of High Dimension Low Sample Size Asymptotics. , 2016, Statistica Sinica.

[4]  Emmanuel J. Candès,et al.  False Discoveries Occur Early on the Lasso Path , 2015, ArXiv.

[5]  Serena Ng,et al.  Working Paper Series , 2019 .

[6]  Chenlei Leng,et al.  High dimensional ordinary least squares projection for screening variables , 2015, 1506.01782.

[7]  Jianqing Fan,et al.  Asymptotics of Empirical Eigen-structure for Ultra-high Dimensional Spiked Covariance Model , 2015, 1502.04733.

[8]  Q. Yao,et al.  High dimensional stochastic regression with latent factors, endogeneity and nonlinearity , 2013, 1310.1990.

[9]  Jonathan E. Taylor,et al.  On model selection consistency of regularized M-estimators , 2013, ArXiv.

[10]  Seung C. Ahn,et al.  Eigenvalue Ratio Test for the Number of Factors , 2013 .

[11]  Jianqing Fan,et al.  Large covariance estimation by thresholding principal orthogonal complements , 2011, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[12]  Clifford Lam,et al.  Factor modeling for high-dimensional time series: inference for the number of factors , 2012, 1206.0613.

[13]  Hansheng Wang,et al.  Factor profiled sure independence screening , 2012 .

[14]  Pascal Sarda,et al.  Factor models and variable selection in high-dimensional regression analysis , 2011 .

[15]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[16]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[17]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[18]  Jianqing Fan,et al.  Sure independence screening in generalized linear models with NP-dimensionality , 2009, The Annals of Statistics.

[19]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[20]  I. Johnstone,et al.  On Consistency and Sparsity for Principal Components Analysis in High Dimensions , 2009, Journal of the American Statistical Association.

[21]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[22]  E. Rio,et al.  A Bernstein type inequality and moderate deviations for weakly dependent sequences , 2009, 0902.0582.

[23]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[24]  Chih-Ling Tsai,et al.  Contour Projected Dimension Reduction , 2008, 0911.3481.

[25]  H. Zou,et al.  One-step Sparse Estimates in Nonconcave Penalized Likelihood Models. , 2008, Annals of statistics.

[26]  S. Geer.  HIGH-DIMENSIONAL GENERALIZED LINEAR MODELS AND THE LASSO , 2008, 0804.0703.

[27]  M. Hallin,et al.  Determining the Number of Factors in the General Dynamic Factor Model , 2007 .

[28]  E. Candès,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[29]  Jianqing Fan,et al.  Sure independence screening for ultrahigh dimensional feature space , 2006, math/0612857.

[30]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[31]  Patrick Warnat,et al.  Customized oligonucleotide microarray gene expression-based classification of neuroblastoma patients outperforms current clinical risk stratification. , 2006, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[32]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[33]  Sydney C. Ludvigson,et al.  Macro Factors in Bond Risk Premia , 2005 .

[34]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[35]  B. Turlach.  DISCUSSION OF "LEAST ANGLE REGRESSION" BY EFRON ET AL. , 2004, math/0406472.

[36]  Jianqing Fan,et al.  Nonconcave penalized likelihood with a diverging number of parameters , 2004, math/0406466.

[37]  Tommaso Proietti,et al.  Forecasting the US unemployment rate , 2003, Comput. Stat. Data Anal..

[38]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[39]  J. Bai,et al.  Inferential Theory for Factor Models of Large Dimensions , 2003 .

[40]  J. Stock,et al.  Forecasting Using Principal Components From a Large Number of Predictors , 2002 .

[41]  Marco Lippi,et al.  The Generalized Dynamic Factor Model , 2002 .

[42]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[43]  J. Bai,et al.  Determining the Number of Factors in Approximate Factor Models , 2000 .

[44]  G. C. Tiao,et al.  Journal of the American Statistical Association Forecasting the U.s. Unemployment Rate Forecasting the U.s. Unemployment Rate , 2022 .

[45]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[46]  Lon-Mu Liu,et al.  FORECASTING AND TIME SERIES ANALYSIS USING THE SCA STATISTICAL SYSTEM , 1994 .

[47]  ByoungSeon Choi,et al.  Arma Model Identification , 1992 .

[48]  G. C. Tiao,et al.  Model Specification in Multivariate Time Series , 1989 .

[49]  G. C. Tiao,et al.  Use of canonical analysis in time series model identification , 1985 .

[50]  Gary Chamberlain,et al.  FUNDS, FACTORS, AND DIVERSIFICATION IN ARBITRAGE PRICING MODELS , 1983 .

[51]  M. Rothschild,et al.  Arbitrage, Factor Structure, and Mean-Variance Analysis on Large Asset Markets , 1982 .

[52]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[53]  George E. P. Box,et al.  Comparison of Forecast and Actuality , 1976 .

[54]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[55]  F. N. David,et al.  FACTOR ANALYSIS AS A STATISTICAL METHOD , 1964 .

[56]  A. E. Maxwell,et al.  Factor Analysis as a Statistical Method. , 1964 .