REGO: Rank-based Estimation of Renyi Information using Euclidean Graph Optimization

We propose a new method for a non- parametric estimation of Renyi and Shan- non information for a multivariate distribu- tion using a corresponding copula, a multi- variate distribution over normalized ranks of the data. As the information of the distri- bution is the same as the negative entropy of its copula, our method estimates this in- formation by solving a Euclidean graph opti- mization problem on the empirical estimate of the distribution's copula. Owing to the properties of the copula, we show that the resulting estimator of Renyi information is strongly consistent and robust. Further, we demonstrate its applicability in image regis- tration in addition to simulated experiments.

[1]  J. Kiefer,et al.  Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator , 1956 .

[2]  R. Prim.  Shortest connection networks and some generalizations , 1957 .

[3]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[4]  D. W. Scott.  On optimal and data based histograms , 1979 .

[5]  Volume Assp,et al.  ACOUSTICS. SPEECH. AND SIGNAL PROCESSING , 1983 .

[6]  J. Steele.  Probability theory and combinatorial optimization , 1987 .

[7]  J. Steele.  Growth Rates of Euclidean Minimal Spanning Trees With Power Weighted Edges , 1988 .

[8]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[9]  I. Vajda.  Theory of statistical inference and information , 1989 .

[10]  P. Massart.  The Tight Constant in the Dvoretzky-Kiefer-Wolfowitz Inequality , 1990 .

[11]  David Eppstein,et al.  Algorithms for Proximity Problems in Higher Dimensions , 1995, Comput. Geom..

[12]  L. Györfi,et al.  Nonparametric entropy estimation. An overview , 1997 .

[13]  J. Yukich.  Probability theory of classical Euclidean optimization problems , 1998 .

[14]  Alfred O. Hero,et al.  Robust entropy estimation strategies based on edge weighted random graphs , 1998, Optics & Photonics.

[15]  A. Hero,et al.  Estimation of Renyi information divergence via pruned minimal spanning trees , 1999, Proceedings of the IEEE Signal Processing Workshop on Higher-Order Statistics. SPW-HOS '99.

[16]  Alfred O. Hero,et al.  Asymptotic theory of greedy approximations to minimal k-point random graphs , 1999, IEEE Trans. Inf. Theory.

[17]  Alfred O. Hero,et al.  Applications of entropic spanning graphs , 2002, IEEE Signal Process. Mag..

[18]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[19]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[20]  J. Yukich,et al.  Statistical Distances Based on Euclidean Graphs , 2005 .

[21]  Jan Kybic.  Incremental Updating of Nearest Neighbor-Based High-Dimensional Entropy Estimation , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[22]  L. Pronzato,et al.  A class of Rényi information estimators for multidimensional densities , 2008, 0810.5302.

[23]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[24]  Barnabás Póczos,et al.  Estimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs , 2010, NIPS.