A class of Rényi information estimators for multidimensional densities

A class of estimators of the Renyi and Tsallis entropies of an unknown distribution f in R^m is presented. These estimators are based on the k-th nearest-neighbor distances computed from a sample of N i.i.d. vectors with distribution f. We show that entropies of any order q, including Shannon's entropy, can be estimated consistently with minimal assumptions on f. Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each.

[1]  H. Lebesgue.  Sur l'intégration des fonctions discontinues , 1910 .

[2]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[3]  J. Doob.  Stochastic processes , 1953 .

[4]  C. Quesenberry,et al.  A nonparametric estimate of a multivariate density function , 1965 .

[5]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1967 .

[6]  Jan Havrda,et al.  Quantification method of classification processes. Concept of structural a-entropy , 1967, Kybernetika.

[7]  Oldrich A Vasicek,et al.  A Test for Normality Based on Sample Entropy , 1976 .

[8]  J. Yackel,et al.  Consistency Properties of Nearest Neighbor Density Function Estimators , 1977 .

[9]  L. Devroye,et al.  The Strong Uniform Consistency of Nearest Neighbor Density Estimates. , 1977 .

[10]  P. Bickel,et al.  Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit Test , 1983 .

[11]  P. Hall.  On powerful distributional tests based on sample spacings , 1986 .

[12]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[13]  J. N. Kapur.  Maximum-entropy models in science and engineering , 1992 .

[14]  David W. Scott,et al.  Multivariate Density Estimation: Theory, Practice, and Visualization , 1992, Wiley Series in Probability and Statistics.

[15]  H. Liero.  A note on the asymptotic behaviour of the distance of the knth nearest neighbour , 1993 .

[16]  P. Hall,et al.  On the estimation of entropy , 1993 .

[17]  A. Tsybakov,et al.  Root-N consistent estimators of entropy for densities with unbounded support , 1994, Proceedings of 1994 Workshop on Information Theory and Statistics.

[18]  Zanette,et al.  Fractal random walks from a variational formalism for Tsallis entropies. , 1994, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[19]  Herman J. Bierens,et al.  Topics in advanced econometrics , 1997 .

[20]  C. Tsallis,et al.  Statistical-mechanical foundation of the ubiquity of Lévy distributions in Nature. , 1995, Physical review letters.

[21]  Tsallis,et al.  Anomalous diffusion in the presence of external forces: Exact time-dependent solutions and their thermostatistical basis. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[22]  J. Yukich,et al.  Asymptotics for Euclidean functionals with power-weighted edges , 1996 .

[23]  M. Basseville.  Information : entropies, divergences et moyennes , 1996 .

[24]  J. Simonoff.  Multivariate Density Estimation , 1996 .

[25]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[26]  K. Zografos,et al.  On maximum entropy characterization of Pearson's type II and VII multivariate distributions , 1999 .

[27]  Alfred O. Hero,et al.  Asymptotic theory of greedy approximations to minimal k-point random graphs , 1999, IEEE Trans. Inf. Theory.

[28]  Andreas Daffertshofer,et al.  Exact time-dependent solutions of the Renyi Fokker–Planck equation and the Fokker–Planck equations related to the entropies proposed by Sharma and Mittal , 2000 .

[29]  Kai-Sheng Song,et al.  Limit theorems for nonparametric sample entropy estimators , 2000 .

[30]  Mathew D. Penrose,et al.  Central limit theorems for k-nearest neighbour distances , 2000 .

[31]  K. Song.  Rényi information, loglikelihood and an intrinsic distribution measure , 2001 .

[32]  Joseph E. Yukich,et al.  Asymptotics for Statistical Distances Based on Voronoi Tessellations , 2002 .

[33]  Antonia J. Jones,et al.  Asymptotic moments of near–neighbour distance distributions , 2002, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[34]  J. Victor.  Binless strategies for estimation of information from neural data. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[35]  Alfred O. Hero,et al.  Applications of entropic spanning graphs , 2002, IEEE Signal Process. Mag..

[36]  Erik G. Miller.  A new class of entropy estimators for multi-dimensional densities , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[37]  John W. Fisher,et al.  ICA Using Spacings Estimates of Entropy , 2003, J. Mach. Learn. Res..

[38]  Alfred O. Hero,et al.  About Closedness by Convolution of the Tsallis Maximizers , 2004 .

[39]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[40]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[41]  Luc Pronzato,et al.  Minimum Entropy Estimation in Semi-Parametric Models: a Candidate for Adaptive Estimation? , 2004 .

[42]  Luc Pronzato,et al.  Minimum-entropy estimation in semi-parametric models , 2005, Signal Process..

[43]  Alfred O. Hero,et al.  Image matching using alpha-entropy measures and entropic graphs , 2005, Signal Process..

[44]  M. N. Goria,et al.  A new class of random vector entropy estimators and its applications in testing statistical hypotheses , 2005 .

[45]  C. Heyde,et al.  Student processes , 2005, Advances in Applied Probability.

[46]  L. Pronzato,et al.  A class of Rényi information estimators for multidimensional densities , 2008, 0810.5302.