Risk of penalized least squares, greedy selection andl 1-penalization for flexible function libraries

For function estimation using penalized squared error criteria, we derive generally applicable risk bounds, showing the balance of accuracy of approximation and penalty relative to the sample size. Attention is given to linear combinations of terms from a given class (such as used in neural network models, projection pursuit regression, function aggregation and multiple linear regression). The risk bounds apply to forward stepwise selection and other relaxed greedy algorithms with penalty on the number of terms, and to ‘1penalized least squares, for which we develop a fast algorithm. 1. Introduction. Flexible regression models are built by combining simple functional forms. Fitting such models to data in a training sample, there is a role for empirical performance criteria such as penalized squared error in selecting components of the function from a given library of candidate terms. With suitable penalty, optimizing the criterion adapts the total weights of combination or the number of components as well as the subset of terms to include. The aim is to produce function estimates which accurately predict responses for new input values with the same distribution as the sample. This generalization capability is characterized by the mean squared error as the statistical risk. In this context, our paper has several interwoven objectives: 1. To analyze performance of penalized least squares estimators with theory of acceptable penalties, such that the estimator optimizing the empirical criterion has risk characterized by a corresponding population property of tradeoff of approximation and penalty relative to the sample size. 2. To allow for flexible function fitting using linear combinations of terms selected from various large or even infinite libraries of functions. 3. To establish that a greedy term selection solves the‘1 penalized squared error problem with bounds on accuracy that compare favorably with competing convex optimization algorithms for large libraries. 4. To demonstrate that different estimators, one based on forward stepwise selection with penalty on the number of terms and another with penalty on the ‘1 norm of coefficients, both achieve approximately the same risk, for target functions that have control on the ‘1 norm of coefficients and for functions in the interpolation classes between these and all of L2.