A Note on Entropy Estimation

We compare an entropy estimator recently discussed by Zhang (2012) with two estimators, and , introduced by Grassberger (2003) and Schürmann (2004). We prove the identity , which has not been taken into account by Zhang (2012). Then we prove that the systematic error (bias) of is less than or equal to the bias of the ordinary likelihood (or plug-in) estimator of entropy. Finally, by numerical simulation, we verify that for the most interesting regime of small sample estimation and large event spaces, the estimator has a significantly smaller statistical error than .

[1]  P. Grassberger.  Entropy Estimates from Insufficient Samplings , 2003, physics/0307138.

[2]  Jonathan W. Pillow,et al.  Bayesian entropy estimation for countable discrete distributions , 2013, J. Mach. Learn. Res..

[3]  Thorsten Gerber,et al.  Handbook Of Mathematical Functions , 2016 .

[4]  William Bialek,et al.  Entropy and Inference, Revisited , 2001, NIPS.

[5]  A. Antos,et al.  Convergence properties of functional estimates for discrete distributions , 2001 .

[6]  R. Rosenfeld.  Nature , 2009, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[7]  Zhiyi Zhang,et al.  Entropy Estimation in Turing's Perspective , 2012, Neural Computation.

[8]  Thomas Schürmann.  Bias analysis in entropy estimation , 2004 .

[9]  Claude E. Shannon,et al.  Prediction and Entropy of Printed English , 1951 .

[10]  Unbiased Estimators for Entropy and Class Number , 2014, 1410.5002.

[11]  Ga Miller,et al.  Note on the bias of information estimates , 1955 .

[12]  Peter Grassberger,et al.  Entropy estimation of symbol sequences. , 1996, Chaos.

[13]  H. Herzel.  Complexity of symbol sequences , 1988 .

[14]  E. H. Simpson.  Measurement of Diversity , 1949, Nature.

[15]  B. Harris.  The Statistical Estimation of Entropy in the Non-Parametric Case , 1975 .

[16]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[17]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[18]  J. Rogers.  Chaos , 1876 .