Nonparametric entropy estimation. An overview

We assume that H(f) is well-defined and is finite. The concept of differential entropy was introduced in Shannon’s original paper ([55]). Since then, entropy has been of great theoretical and applied interest. The basic properties ∗This research was supported by the Scientific Exchange Program between the Belgian Academy of Sciences and the Hungarian Academy of Sciences in the field of Mathematical Information Theory, and NATO Research Grant No. CRG 931030.

[1]  M GREENWOOD,et al.  The statistical study of infectious diseases. , 1946, Journal of the Royal Statistical Society. Series A.

[2]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[3]  THE RANDOM DIVISION OF AN INTERVAL-PART III , 1951 .

[4]  D. Darling.  On a Class of Problems Related to the Random Division of an Interval , 1953 .

[5]  A. Rényi.  On the dimension and entropy of probability distributions , 1959 .

[6]  Jaroslav Kožešnk,et al.  Information Theory, Statistical Decision Functions, Random Processes , 1962 .

[7]  Herbert Gish,et al.  Asymptotically efficient quantizing , 1968, IEEE Trans. Inf. Theory.

[8]  F. P. Tarasenko.  On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable, and the distribution-free entropy test of goodness-of-fit , 1968 .

[9]  Yuriy G. Dmitriev,et al.  On the Estimation of Functionals of the Probability Density and Its Derivatives , 1974 .

[10]  J. Ord,et al.  Characterization Problems in Mathematical Statistics , 1975 .

[11]  Oldrich A Vasicek,et al.  A Test for Normality Based on Sample Entropy , 1976 .

[12]  Ibrahim A. Ahmad,et al.  A nonparametric estimation of the entropy for absolutely continuous distributions (Corresp.) , 1976, IEEE Trans. Inf. Theory.

[13]  Noel A Cressie,et al.  On the logarithms of high-order spacings , 1976 .

[14]  Noel A Cressie,et al.  The minimum of higher order gaps , 1977 .

[15]  N. Cressie.  Power results for tests based on high-order gaps , 1978 .

[16]  Pushpa N. Rathie,et al.  On the entropy of continuous probability distributions (Corresp.) , 1978, IEEE Trans. Inf. Theory.

[17]  Allen Gersho,et al.  Asymptotically optimal block quantization , 1979, IEEE Trans. Inf. Theory.

[18]  Henri Theil,et al.  The entropy of the maximum entropy distribution , 1980 .

[19]  Edward C. van der Meulen,et al.  Entropy-Based Tests of Uniformity , 1981 .

[20]  Prakasa Rao.  Nonparametric functional estimation , 1983 .

[21]  D. Gokhale.  On entropy-based goodness-of-fit tests , 1983 .

[22]  Russell C. H. Cheng,et al.  Estimating Parameters in Continuous Univariate Distributions with a Shifted Origin , 1983 .

[23]  Meulen Edward C. van der,et al.  Entropy-Based Statistical Inference, Ii: Selection-Of-The-Best/Complete Ranking For Continuous Distributions On (0,1), With Applications To Random Number Generators , 1983 .

[24]  P. Bickel,et al.  Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit Test , 1983 .

[25]  P. Hall.  Limit theorems for sums of general functions of m-spacings , 1984, Mathematical Proceedings of the Cambridge Philosophical Society.

[26]  Jan Beirlant,et al.  The empirical distribution function and strong laws for functions of order statistics of uniform spacings , 1985 .

[27]  P. Hall.  On powerful distributional tests based on sample spacings , 1986 .

[28]  John Van Ryzin,et al.  Large sample properties of maximum entropy histograms , 1986, IEEE Trans. Inf. Theory.

[29]  Denzil G. Fiebig,et al.  Exploiting continuity : maximum entropy estimation of continuous distributions , 1986 .

[30]  E. Dudewicz,et al.  On the Power of Entropy-Based Tests Against Bump-Type Alternatives , 1987 .

[31]  L. Györfi,et al.  Density-free convergence properties of various estimators of entropy , 1987 .

[32]  P. Hall.  On Kullback-Leibler loss and density estimation , 1987 .

[33]  D. V. Gokhale,et al.  Entropy expressions and their estimators for multivariate distributions , 1989, IEEE Trans. Inf. Theory.

[34]  Abdelkader Mokkadem,et al.  Estimation of the entropy and information of absolutely continuous random variables , 1989, IEEE Trans. Inf. Theory.

[35]  Denzil G. Fiebig,et al.  Exploiting Continuity: Maximum Entropy Estimation of Continuous Distributions , 1989 .

[36]  G. S. Mudholkar,et al.  EMPIRICALLY ACCELERATED CONVERGENCE OF THE SPACINGS STATISTICS TO NORMALITY , 1990 .

[37]  Emanuel Parzen,et al.  Goodness of Fit Tests and Entropy , 1990 .

[38]  E. Dudewicz,et al.  Maximum entropy methods in modern spectroscopy: a review and an empiric entropy approach , 1991 .

[39]  P. Robinson.  Consistent Nonparametric Entropy-Based Testing , 1991 .

[40]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[41]  Nonparametric Entropy Estimation Based On Randomly Censored Data , 1991, Proceedings. 1991 IEEE International Symposium on Information Theory.

[42]  P. Hall,et al.  On the estimation of entropy , 1993 .

[43]  A. Tsybakov,et al.  Root-N consistent estimators of entropy for densities with unbounded support , 1994, Proceedings of 1994 Workshop on Information Theory and Statistics.

[44]  Tamás Linder,et al.  On the asymptotic tightness of the Shannon lower bound , 1994, IEEE Trans. Inf. Theory.

[45]  Tamás Linder,et al.  Asymptotic entropy-constrained performance of tessellating and universal randomized lattice quantization , 1994, IEEE Trans. Inf. Theory.

[46]  Marjorie G. Hahn,et al.  Limit theorems for the logarithm of sample spacings , 1995 .

[47]  Edward C. van der Meulen,et al.  Entropy-Based Random Number Evaluation , 1995 .