BOOTSTRAP-BASED PENALTY CHOICE FOR THE LASSO , ACHIEVING ORACLE PERFORMANCE

In theory, if penalty parameters are chosen appropriately then the lasso can eliminate unnecessary variables in prediction problems, and improve the performance of predictors based on the variables that remain. However, standard methods for tuning-parameter choice, for example techniques based on the bootstrap or cross-validation, are not sufficiently accurate to achieve this level of precision. Until Zou’s (2006) proposal for an inversely-weighted lasso, this difficulty led to speculation that it might not be possible to achieve oracle performance using the lasso. In the present paper we show that a straightforward application of the m-out-of-n bootstrap produces adaptive penalty estimates that confer oracle properties on the lasso. The application is of interest in its own right since, unlike many uses of the m-out-of-n bootstrap, it is not designed to estimate a non-normal distribution; the limiting distributions of regression parameter estimators are normal. Instead, the m-out-of-n bootstrap overcomes the tendency of the standard bootstrap to confound the errors committed in determining whether or not a parameter value is zero, with estimation errors for nonzero parameters.