MaxEnt Upper Bounds for the Differential Entropy of Univariate Continuous Distributions

We present a series of closed-form upper bounds of the differential entropy of univariate continuous distributions based on the maximum entropy principle. We apply those bounds to Gaussian mixture models, and study their tightness properties.

[1]  Frank Nielsen,et al.  Statistical exponential families: A digest with flash cards , 2009, ArXiv.

[2]  M. H. Protter,et al.  THE SOLUTION OF THE PROBLEM OF INTEGRATION IN FINITE TERMS , 1970 .

[3]  Frank Nielsen,et al.  Guaranteed Bounds on the Kullback–Leibler Divergence of Univariate Mixtures , 2016, IEEE Signal Processing Letters.

[4]  Keisuke Yamazaki,et al.  Kullback Information of Normal Mixture is not an Analytic Function , 2004 .

[5]  Frank Nielsen,et al.  A series of maximum entropy upper bounds of the differential entropy , 2016, ArXiv.

[6]  Frank Nielsen,et al.  Entropies and cross-entropies of exponential families , 2010, 2010 IEEE International Conference on Image Processing.

[7]  Lucy Y. Pao,et al.  Variance estimation and ranking of Gaussian mixture distributions in target tracking applications , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[8]  J. E. Glynn,et al.  Numerical Recipes: The Art of Scientific Computing , 1989 .

[9]  J. T. Spooner,et al.  Adaptive and Learning Systems for Signal Processing, Communications, and Control , 2006 .

[10]  J. V. Michalowicz,et al.  Handbook of Differential Entropy , 2013 .

[11]  Frank Nielsen,et al.  Guaranteed Bounds on Information-Theoretic Measures of Univariate Mixtures Using Piecewise Log-Sum-Exp Inequalities , 2016, Entropy.

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  Kamyar Moshksar,et al.  Arbitrarily Tight Bounds on Differential Entropy of Gaussian Mixtures , 2016, IEEE Transactions on Information Theory.

[14]  Manuel Bronstein,et al.  Symbolic integration I: transcendental functions , 1997 .

[15]  A. Mohammad-Djafari.  A Matlab Program to Calculate the Maximum Entropy Distributions , 2001, physics/0111126.

[16]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[17]  L. Brown.  Fundamentals of statistical exponential families: with applications in statistical decision theory , 1986 .

[18]  Will Tribbey,et al.  Numerical Recipes: The Art of Scientific Computing (3rd Edition) is written by William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery, and published by Cambridge University Press, © 2007, hardback, ISBN 978-0-521-88068-8, 1235 pp. , 1987, SOEN.

[19]  Moshe Shared,et al.  On Mixtures from Exponential Families , 1980 .

[20]  Frank Nielsen,et al.  A closed-form expression for the Sharma–Mittal entropy of exponential families , 2011, ArXiv.

[21]  James H. Davenport.  Complexity of Integration, Special Values, and Recent Developments , 2016, ICMS.

[22]  Jonathan M. Nichols,et al.  Calculation of Differential Entropy for a Mixed Gaussian Distribution , 2008, Entropy.