On Improved Loss Estimation for Shrinkage Estimators

Let X be a random vector with distribution Pθ where θ is an unknown parameter. When estimating θ by some estimator φ(X) under a loss function L(θ, φ), classical decision theory advocates that such a decision rule should be used if it has suitable properties with respect to the frequentist risk R(θ, φ). However, after having observed X = x, instances arise in practice in which φ is to be accompanied by an assessment of its loss, L(θ, φ(x)), which is unobservable since θ is unknown. A common approach to this assessment is to consider estimation of L(θ, φ(x)) by an estimator δ, called a loss estimator. We present an expository development of loss estimation with substantial emphasis on the setting where the distributional context is normal and its extension to the case where the underlying distribution is spherically symmetric. Our overview covers improved loss estimators for least squares but primarily focuses on shrinkage estimators. Bayes estimation is also considered and comparisons are made with unbiased estimation.

[1]  C. Lele.  INADMISSIBILITY OF LOSS ESTIMATORS , 1992 .

[2]  M. Wells,et al.  Estimation of a Loss Function for Spherically Symmetric Distributions in the General Linear Model , 1995 .

[3]  J. Kiefer.  Admissibility of Conditional Confidence Procedures , 1976 .

[4]  L. R. Haff.  An identity for the Wishart distribution with applications , 1979 .

[5]  Martin T. Wells,et al.  Robust shrinkage estimation for elliptically symmetric distributions with unknown covariance matrix , 2003 .

[6]  C. Mallows.  Some Comments on Cp , 2000, Technometrics.

[7]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[8]  Estimation of Normal Means: Frequentist Estimation of Loss , 1989 .

[9]  N. D. Plessis.  An introduction to potential theory , 1970 .

[10]  C. Lele.  Admissibility Results in Loss Estimation , 1993 .

[11]  D. Dey,et al.  Estimation of a covariance matrix under Stein's loss , 1985 .

[12]  Else Sandved.  Ancillary Statistics and Estimation of the Loss in Estimation Problems , 1968 .

[13]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[14]  D. Cellier,et al.  Shrinkage Estimators under Spherical Symmetry for the General Linear Model , 1995 .

[15]  On Inadmissibility of Some Unbiased Estimates of Loss , 1988 .

[16]  M. Bock.  Shrinkage Estimators: Pseudo-Bayes Rules for Normal Mean Vectors , 1988 .

[17]  R. Tibshirani,et al.  On the “degrees of freedom” of the lasso , 2007, 0712.0881.

[18]  A. Takemura.  An orthogonally invariant minimax estimator of the covariance matrix of a multivariate normal population , 1984 .

[19]  On unbiased and improved loss estimation for the mean of a multivariate normal distribution with unknown variance , 2004 .

[20]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[21]  Lawrence D. Brown,et al.  INADMISSIBILITY OF THE USUAL ESTIMATORS OF SCALE PARAMETERS IN PROBLEMS WITH UNKNOWN LOCATION AND SCALE PARAMETERS , 1968 .

[22]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[23]  Tatsuya Kubokawa,et al.  Estimating the covariance matrix: a new approach , 2003 .

[24]  J. Kiefer.  Conditional Confidence Statements and Confidence Estimators , 1977 .

[25]  S. Kotz,et al.  Symmetric Multivariate and Related Distributions , 1989 .

[26]  Dominique Fourdrinier,et al.  On Bayes and unbiased estimators of loss , 2003 .

[27]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[28]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[29]  P. Massart,et al.  Risk bounds for model selection via penalization , 1999 .

[30]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[31]  E. Lehmann,et al.  Completeness, Similar Regions, and Unbiased Estimation—Part II , 2012 .

[32]  J. Zidek,et al.  Simultaneous Estimation of the Means of Independent Poisson Laws , 1975 .

[33]  C. Robert,et al.  Robust shrinkage estimators of the location parameter for elliptically symmetric distributions , 1989 .

[34]  B. Efron.  The Estimation of Prediction Error , 2004 .

[35]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[36]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[37]  Peter L. Bartlett,et al.  Model Selection and Error Estimation , 2000, Machine Learning.

[38]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[39]  Estimating a general function of a quadratic function , 2008 .

[40]  C. Stein.  Inadmissibility of the usual estimator for the variance of a normal distribution with unknown mean , 1964 .

[41]  L. Brown.  Admissible Estimators, Recurrent Diffusions, and Insoluble Boundary Value Problems , 1971 .

[42]  A. Rukhin.  Estimated Loss and Admissible Loss Estimators , 1988 .

[43]  Tatsuya Kubokawa,et al.  Robust improvement in estimation of a covariance matrix in an elliptically contoured distribution , 1999 .

[44]  Jianming Ye.  On Measuring and Correcting the Effects of Data Mining and Model Selection , 1998 .

[45]  H. Hudson.  A Natural Identity for Exponential Families with Applications in Multiparameter Estimation , 1978 .