Efficient multivariate entropy estimation via $k$-nearest neighbour distances

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient and achieve the local asymptotic minimax lower bound with respect to squared error loss. To this end, we study weighted averages of the estimators originally proposed by Kozachenko and Leonenko (1987), based on the $k$-nearest neighbour distances of a sample of $n$ independent and identically distributed random vectors in $\mathbb{R}^d$. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness, while the original unweighted estimator is typically only efficient when $d \leq 3$. In addition to the new estimator proposed and theoretical understanding provided, our results facilitate the construction of asymptotically valid confidence intervals for the entropy of asymptotically minimal width.

[1]  Oldrich A Vasicek,et al.  A Test for Normality Based on Sample Entropy , 1976 .

[2]  Noel A Cressie,et al.  On the logarithms of high-order spacings , 1976 .

[3]  J. Wellner,et al.  Empirical Processes with Applications to Statistics , 2009 .

[4]  Ludwig Paditz,et al.  On the analytical structure of the constant in the nonuniform version of the esseen inequality , 1989 .

[5]  F. Götze.  On the Rate of Convergence in the Multivariate CLT , 1991 .

[6]  I. Ibragimov,et al.  Asymptotically normal families of distributions and efficient estimation , 1991 .

[7]  P. Hall,et al.  On the estimation of entropy , 1993 .

[8]  A. Tsybakov,et al.  Root-N consistent estimators of entropy for densities with unbounded support , 1994, Proceedings of 1994 Workshop on Information Theory and Statistics.

[9]  B. Laurent.  Efficient estimation of integral functionals of a density , 1996 .

[10]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[11]  A. Nemirovski,et al.  On estimation of the Lr norm of a regression function , 1999 .

[12]  Chong-Ho Choi,et al.  Input Feature Selection by Mutual Information Based on Parzen Window , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Harshinder Singh,et al.  Nearest Neighbor Estimates of Entropy , 2003 .

[14]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[15]  John W. Fisher,et al.  ICA Using Spacings Estimates of Entropy , 2003, J. Mach. Learn. Res..

[16]  Louis H. Y. Chen,et al.  An Introduction to Stein's Method , 2005 .

[17]  M. N. Goria,et al.  A new class of random vector entropy estimators and its applications in testing statistical hypotheses , 2005 .

[18]  Liam Paninski,et al.  Undersmoothed Kernel Entropy Estimators , 2008, IEEE Transactions on Information Theory.

[19]  Neeraj Misra,et al.  Kn-nearest neighbor estimators of entropy , 2008 .

[20]  Yu. Golubev,et al.  On entropy estimation by m-spacing method , 2009 .

[21]  T. Cai,et al.  Testing composite hypotheses, Hermite polynomials and optimal estimation of a nonsmooth functional , 2011, 1105.3039.

[22]  Alfred O. Hero,et al.  Ensemble Estimators for Multivariate Entropy Estimation , 2013, IEEE Transactions on Information Theory.

[23]  Luc Devroye,et al.  Lectures on the Nearest Neighbor Method , 2015 .

[24]  S. Delattre,et al.  On the Kozachenko-Leonenko entropy estimator , 2016, 1602.07440.

[25]  Barnabás Póczos,et al.  Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation , 2016, ArXiv.

[26]  Francis Edward Su,et al.  On Choosing and Bounding Probability , 2016 .

[27]  Kevin R. Moon,et al.  Nonparametric Ensemble Estimation of Distributional Functionals , 2016 .

[28]  Pramod Viswanath,et al.  Demystifying fixed k-nearest neighbor information estimators , 2016, 2017 IEEE International Symposium on Information Theory (ISIT).