Estimating Differential Entropy using Recursive Copula Splitting

A method for estimating the Shannon differential entropy of multidimensional random variables using independent samples is described. The method is based on decomposing the distribution into a product of marginal distributions and joint dependency, also known as the copula. The entropy of marginals is estimated using one-dimensional methods. The entropy of the copula, which always has a compact support, is estimated recursively by splitting the data along statistically dependent dimensions. The method can be applied both for distributions with compact and non-compact supports, which is imperative when the support is not known or of a mixed type (in different dimensions). At high dimensions (larger than 20), numerical examples demonstrate that our method is not only more accurate, but also significantly more efficient than existing approaches.

[1]  Igor Vajda,et al.  Estimation of the Information by an Adaptive Partitioning of the Observation Space , 1999, IEEE Trans. Inf. Theory.

[3]  Dan Stowell,et al.  Fast Multidimensional Entropy Estimation by $k$-d Partitioning , 2009, IEEE Signal Processing Letters.

[4]  P. Embrechts,et al.  STATISTICAL INFERENCE FOR COPULAS IN HIGH DIMENSIONS: A SIMULATION STUDY , 2013, ASTIN Bulletin.

[5]  A. Hero,et al.  Empirical estimation of entropy functionals with confidence , 2010, 1012.4188.

[6]  Jie Sun,et al.  Geometric k-nearest neighbor estimation of entropy and mutual information , 2017, Chaos.

[7]  Laura Sacerdote,et al.  Non-Parametric Estimation of Mutual Information through the Entropy of the Linkage , 2013, Entropy.

[8]  Pramod Viswanath,et al.  Density functional estimators with k-nearest neighbor bandwidths , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[9]  Erik G. Miller.  A new class of entropy estimators for multi-dimensional densities , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[10]  Han Liu,et al.  Distribution-free tests of independence in high dimensions , 2014, Biometrika.

[11]  Alfred O. Hero,et al.  Ensemble Estimators for Multivariate Entropy Estimation , 2013, IEEE Transactions on Information Theory.

[12]  K. Knuth.  Optimal Data-Based Binning for Histograms , 2006, physics/0605197.

[13]  Ram Avinery,et al.  Universal and Accessible Entropy Estimation Using a Compression Algorithm. , 2017, Physical review letters.

[14]  Piotr Indyk,et al.  Simultaneous Nearest Neighbor Search , 2016, SoCG.

[15]  Fabrizio Durante,et al.  Copula Theory and Its Applications , 2010 .

[16]  Yoav Y. Schechner,et al.  Fast kernel entropy estimation and optimization , 2005, Signal Process..

[17]  Driss Aboutajdine,et al.  Textural feature selection by joint mutual information based on Gaussian mixture model for multispectral image classification , 2010, Pattern Recognit. Lett..

[18]  Jacob Goldberger,et al.  ICA based on a Smooth Estimation of the Differential Entropy , 2008, NIPS.

[19]  Ram Avinery,et al.  Universal and efficient entropy estimation using a compression algorithm , 2017 .

[20]  Renato Vicente,et al.  An information-theoretic approach to statistical dependence: Copula information , 2009, ArXiv.

[21]  Vijay P. Singh,et al.  Integrating Entropy and Copula Theories for Hydrologic Modeling and Analysis , 2015, Entropy.

[22]  Pramod Viswanath,et al.  Breaking the Bandwidth Barrier: Geometrical Adaptive Entropy Estimation , 2016, IEEE Transactions on Information Theory.

[23]  Deniz Erdogmus,et al.  Continuously Differentiable Sample-Spacing Entropy Estimation , 2008, IEEE Transactions on Neural Networks.

[24]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[25]  Dov Levine,et al.  Quantifying Hidden Order out of Equilibrium , 2017, Physical Review X.

[26]  H. Joe.  Estimation of entropy and other functionals of a multivariate density , 1989 .

[27]  P. Strevens.  Iii , 1985 .

[28]  C. Sempi,et al.  Copula Theory: An Introduction , 2010 .

[29]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[30]  Harshinder Singh,et al.  Nearest Neighbor Estimates of Entropy , 2003 .

[31]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[32]  Chong-Ho Choi,et al.  Input Feature Selection by Mutual Information Based on Parzen Window , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  C. Granger,et al.  USING THE MUTUAL INFORMATION COEFFICIENT TO IDENTIFY LAGS IN NONLINEAR MODELS , 1994 .

[34]  Kai Yu,et al.  Feature Selection for Gene Expression Using Model-Based Entropy , 2010, IEEE/ACM Transactions on Computational Biology and Bioinformatics.