Bootstrap tuning in ordered model selection

In the problem of model selection for a given family of linear estimators, ordered by their variance, we offer a new "smallest accepted" approach motivated by Lepski's method and multiple testing theory. The procedure selects the smallest model which satisfies an acceptance rule based on comparison with all larger models. The method is completely data-driven and does not use any prior information about the variance structure of the noise: its parameters are adjusted to the underlying possibly heterogeneous noise by the so-called "propagation condition" using a wild bootstrap method. The validity of the bootstrap calibration is proved for finite samples with an explicit error bound. We provide a comprehensive theoretical study of the method and describe in detail the set of possible values of the selector \( \hat{m} \). We also establish some precise oracle error bounds for the corresponding estimator \( \hat{\theta} = \tilde{\theta}_{\hat{m}} \) which equally applies to estimation of the whole parameter vectors, some subvector or linear mapping, as well as the estimation of a linear functional.

[1]  P. Massart,et al.  Concentration inequalities and model selection , 2007 .

[2]  A. Tsybakov.  On the best rate of adaptive estimation in some inverse problems , 2000 .

[3]  A. Tsybakov,et al.  Oracle inequalities for inverse problems , 2002 .

[4]  N. Weber,et al.  Discussion: Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[5]  L. Cavalier,et al.  Risk hull method and regularization by projections of ill-posed inverse problems , 2006, math/0611228.

[6]  P. Massart,et al.  Adaptive estimation of a quadratic functional by model selection , 2000 .

[7]  Weining Wang,et al.  Local Quantile Regression , 2010, 1208.5384.

[8]  T. Tony Cai,et al.  A note on nonparametric estimation of linear functionals , 2003 .

[9]  V. Spokoiny.  Parametric estimation. Finite sample theory , 2011, 1111.3029.

[10]  R. Nickl,et al.  CONFIDENCE BANDS IN DENSITY ESTIMATION , 2010, 1002.4801.

[11]  A. Kneip.  Ordered Linear Smoothers , 1994 .

[12]  Richard Nickl,et al.  Spatially adaptive density estimation by localised Haar projections , 2011, 1111.2807.

[13]  P. Massart,et al.  Risk bounds for model selection via penalization , 1999 .

[14]  T. Tony Cai,et al.  On Adaptive Estimation of Linear Functionals , 2005 .

[15]  L. Birge,et al.  An alternative point of view on Lepski's method , 2001 .

[16]  E. Mammen,et al.  Optimal spatial adaptation to inhomogeneous smoothness: an approach based on kernel estimates with variable bandwidth selectors , 1997 .

[17]  O. Lepskii,et al.  Asymptotically minimax adaptive estimation. II: Schemes without optimal adaptation: adaptive estimators , 1993 .

[18]  A. Goldenshluger.  A universal procedure for aggregating estimators , 2007, 0704.2500.

[19]  Sylvain Arlot.  Model selection by resampling penalization , 2007, 0906.3124.

[20]  O. Lepskii.  Asymptotically Minimax Adaptive Estimation. I: Upper Bounds. Optimally Adaptive Estimates , 1992 .

[21]  Victor Chernozhukov,et al.  Anti-concentration and honest, adaptive confidence bands , 2013 .

[22]  Vladimir Spokoiny,et al.  Parameter tuning in pointwise adaptation using a propagation approach , 2009, 0908.3596.

[23]  Changbao Wu,et al.  Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis , 1986 .

[24]  A. Dalalyan,et al.  Sharp Oracle Inequalities for Aggregation of Affine Estimators , 2011, 1104.3969.

[25]  E. Mammen.  Bootstrap and Wild Bootstrap for High Dimensional Linear Models , 1993 .

[26]  P. Massart,et al.  Minimal Penalties for Gaussian Model Selection , 2007 .

[27]  O. Lepskii.  On a Problem of Adaptive Estimation in Gaussian White Noise , 1991 .