Information theory and superefficiency

The asymptotic risk of efficient estimators with Kullback-Leibler loss in smoothly parametrized statistical models is k/2n, where k is the parameter dimension and n is the sample size. Under fairly general conditions, we given a simple information-theoretic proof that the set of parameter values where any arbitrary estimator is superefficient is negligible. The proof is based on a result of Rissanen that codes have asymptotic redundancy not smaller than (k/2)log n, except in a set of measure 0.

[1]  H. Jeffreys.  An invariant form for the prior probability in estimation problems , 1946, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[2]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[3]  Le Cam,et al.  On some asymptotic properties of maximum likelihood estimates and related Bayes' estimates , 1953 .

[4]  L. Goddard.  Information Theory , 1962, Nature.

[5]  R. R. Bahadur.  On Fisher's Bound for Asymptotic Variances , 1964 .

[6]  G. Lorentz.  Metric entropy and approximation , 1966 .

[7]  S. M. Ali,et al.  A General Class of Coefficients of Divergence of One Distribution from Another , 1966 .

[8]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[9]  R. R. Bahadur.  Rates of Convergence of Estimates and Test Statistics , 1967 .

[10]  R. R. Bahadur.  1. Some Limit Theorems in Statistics , 1971 .

[11]  R. Z. Khasʹminskiĭ,et al.  Statistical estimation : asymptotic theory , 1981 .

[12]  H. Heyer,et al.  Information and Sufficiency , 1982 .

[13]  D. Lindley.  Bayes theory , 1984 .

[14]  Jorma Rissanen,et al.  Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.

[15]  L. Birge,et al.  On estimating a density using Hellinger distance and some other strange facts , 1986 .

[16]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[17]  L. L. Cam,et al.  Asymptotic Methods In Statistical Decision Theory , 1986 .

[18]  A. Barron.  Are Bayes Rules Consistent in Information , 1987 .

[19]  Andrew R. Barron,et al.  Information-theoretic asymptotics of Bayes methods , 1990, IEEE Trans. Inf. Theory.

[20]  Andrew R. Barron,et al.  Minimum complexity density estimation , 1991, IEEE Trans. Inf. Theory.

[21]  V. Vovk.  Asymptotic efficiency of estimators : algorithmic approach , 1992 .

[22]  Lawrence D. Brown.  An Information Inequality for the Bayes Risk Under Truncated Squared Error Loss , 1993 .

[23]  Neri Merhav,et al.  A strong version of the redundancy-capacity theorem of universal coding , 1995, IEEE Trans. Inf. Theory.

[24]  F. Komaki.  On asymptotic properties of predictive distributions , 1996 .

[25]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[26]  Lawrence D. Brown,et al.  Superefficiency in Nonparametric Function Estimation , 1997 .

[27]  J. Hartigan.  The maximum likelihood prior , 1998 .

[28]  Yuhong Yang,et al.  Information-theoretic determination of minimax rates of convergence , 1999 .

[29]  N. Čencov.  Statistical Decision Rules and Optimal Inference , 2000 .