Understanding Shrinkage Estimators: From Zero to Oracle to James-Stein

The standard estimator of the population mean is the sample mean, which is unbiased. Constructing an estimator by shrinking the sample mean results in a biased estimator, with an expected value less than the population mean. On the other hand, shrinkage always reduces the estimator's variance and can reduce its mean squared error. This paper tries to explain how that works. I start with estimating a single mean using the zero estimator and the oracle estimator, and continue with the grand-mean estimator. Thus prepared, it is easier to understand the James-Stein estimator, in its simple form with known homogeneous variance and in extensions. The James-Stein estimator combines the oracle estimate's coefficient shrinking with the grand mean estimator's cancelling out of overestimates and underestimates.

[1]  Bruce E. Hansen,et al.  The Risk of James–Stein and Lasso Shrinkage , 2016 .

[2]  William E. Strawderman,et al.  Stein Estimation for Spherically Symmetric Distributions: Recent Developments , 2012, 1203.4724.

[3]  T. Kubokawa.  A Unified Approach to Improving Equivariant Estimators , 1994 .

[4]  Stephen M. Stigler,et al.  The 1988 Neyman Memorial Lecture: A Galtonian Perspective on Shrinkage Estimators , 1990 .

[5]  George G. Judge,et al.  A simple form for the inverse moments of non-central χ2 andF random variables and certain confluent hypergeometric functions , 1984 .

[6]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[7]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[8]  James O. Berger,et al.  Improving on Inadmissible Estimators in Continuous Exponential Families with Applications to Simultaneous Estimation of Gamma Scale Parameters , 1980 .

[9]  B. Efron,et al.  Stein's Paradox in Statistics , 1977 .

[10]  B. Efron.  Biased Versus Unbiased Estimation , 1975 .

[11]  E. Fama,et al.  Risk, Return, and Equilibrium: Empirical Tests , 1973, Journal of Political Economy.

[12]  B. Efron,et al.  Stein's Estimation Rule and Its Competitors- An Empirical Bayes Approach , 1973 .

[13]  L. Brown.  On the Admissibility of Invariant Estimators of One or More Location Parameters , 1966 .

[14]  D. Blackwell.  On the Translation Parameter Problem for Discrete Variables , 1951 .

[15]  C. Srinivasan,et al.  Stein's Phenomenon , 2011 .

[16]  Michel Daoud Yacoub,et al.  The multivariate α-μ distribution , 2010, IEEE Transactions on Wireless Communications.

[17]  J. Neyman,et al.  INADMISSIBILITY OF THE USUAL ESTIMATOR FOR THE MEAN OF A MULTIVARIATE NORMAL DISTRIBUTION , 2005 .

[18]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .