Model selection and sharp asymptotic minimaxity

We obtain sharp minimax results for estimation of an n-dimensional normal mean under quadratic loss. The estimators are chosen by penalized least squares with a penalty that grows like ck log(n/k), for k equal to the number of nonzero elements in the estimating vector. For a wide range of sparse parameter spaces, we show that the penalized estimator achieves the exact minimax rate with the correct multiplication constant if and only if c equals 2. Our results unify the theory obtained by many other authors for penalized estimation of normal means. In particular we establish that a conjecture by Abramovich et al. (Ann Stat 34:584–653, 2006) is true.

[1]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[2]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[3]  Dharmendra S. Modha,et al.  Memory-Universal Prediction of Stationary Random Processes , 1998, IEEE Trans. Inf. Theory.

[4]  D. Mason,et al.  Central limit theorems for sums of extreme values , 1985, Mathematical Proceedings of the Cambridge Philosophical Society.

[5]  P. Massart,et al.  Minimal Penalties for Gaussian Model Selection , 2007 .

[6]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[7]  I. Johnstone,et al.  Minimax estimation via wavelet shrinkage , 1998 .

[8]  S. Gupta,et al.  Statistical decision theory and related topics IV , 1988 .

[9]  Harrison H. Zhou,et al.  Asymptotic equivalence of spectral density estimation and gaussian white noise , 2009, 0903.1314.

[10]  T. Tony Cai,et al.  Nonparametric regression in exponential families , 2010, 1010.3836.

[11]  P. Massart,et al.  Risk bounds for model selection via penalization , 1999 .

[12]  I. Johnstone,et al.  Adapting to unknown sparsity by controlling the false discovery rate , 2005, math/0505374.

[13]  H. Akaike.  A new look at the statistical model identification , 1974 .

[14]  I. Johnstone,et al.  Minimax Risk over l p-Balls for l q-error , 1994 .

[15]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[16]  Yuhong Yang.  MODEL SELECTION FOR NONPARAMETRIC REGRESSION , 1997 .

[17]  Herbert A. David,et al.  Order Statistics, Third Edition , 2003, Wiley Series in Probability and Statistics.

[18]  Yuhong Yang,et al.  An Asymptotic Property of Model Selection Criteria , 1998, IEEE Trans. Inf. Theory.

[19]  Dean P. Foster,et al.  Local Asymptotic Coding and the Minimum Description Length , 1999, IEEE Trans. Inf. Theory.

[20]  H. N. Nagaraja,et al.  Order Statistics, Third Edition , 2005, Wiley Series in Probability and Statistics.

[21]  P. Massart,et al.  Gaussian model selection , 2001 .

[22]  L. Brown,et al.  Asymptotic equivalence of nonparametric regression and white noise , 1996 .

[23]  T. Tony Cai,et al.  Asymptotic Equivalence and Adaptive Estimation for Robust Nonparametric Regression , 2009, 0909.0343.

[24]  Felix Abramovich,et al.  On optimality of Bayesian testimation in the normal means problem , 2007, 0712.0904.

[25]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[26]  B. Efron.  Robbins, Empirical Bayes, And Microarrays , 2001 .

[27]  I. Johnstone.  Minimax Bayes, Asymptotic Minimax and Sparse Wavelet Priors , 1994 .

[28]  M. Nussbaum.  Asymptotic Equivalence of Density Estimation and Gaussian White Noise , 1996 .

[29]  I. Johnstone,et al.  Maximum Entropy and the Nearly Black Object , 1992 .

[30]  Y. Benjamini,et al.  A simple forward selection procedure based on false discovery rate control , 2009, 0905.2819.

[31]  Dean Phillips Foster,et al.  Calibration and Empirical Bayes Variable Selection , 1997 .

[32]  R. Tibshirani,et al.  The Covariance Inflation Criterion for Adaptive Model Selection , 1999 .

[33]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[34]  I. Johnstone,et al.  Minimax risk overlp-balls forlp-error , 1994 .

[35]  Harrison H. Zhou,et al.  Robust nonparametric estimation via wavelet median regression , 2008, 0810.4802.

[36]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[37]  C. Shihong.  Large deviation theorem for Hill's estimator , 1992 .