On convergence properties of Shannon entropy

Convergence properties of Shannon entropy are studied. In the differential setting, it is known that weak convergence of probability measures (convergence in distribution) is not sufficient for convergence of the associated differential entropies. In that direction, an interesting example is introduced and discussed in light of new general results provided here for the desired differential entropy convergence, which take into account both compactly and uncompactly supported densities. Convergence of differential entropy is also characterized in terms of the Kullback-Liebler discriminant for densities with fairly general supports, and it is shown that convergence in variation of probability measures guarantees such convergence under an appropriate boundedness condition on the densities involved. Results for the discrete setting are also provided, allowing for infinitely supported probability measures, by taking advantage of the equivalence between weak convergence and convergence in variation in that setting.

[1]  O. Kallenberg.  Foundations of Modern Probability , 2021, Probability Theory and Stochastic Modelling.

[2]  David Williams,et al.  Probability with Martingales , 1991, Cambridge mathematical textbooks.

[3]  P. Hall,et al.  On the estimation of entropy , 1993 .

[4]  Raymond W. Yeung,et al.  On the Discontinuity of the Shannon Information Measures , 2005, IEEE Transactions on Information Theory.

[5]  A. Barron.  ENTROPY AND THE CENTRAL LIMIT THEOREM , 1986 .

[6]  Thomas M. Cover,et al.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) , 2006 .

[7]  Bert Fristedt,et al.  A modern approach to probability theory , 1996 .

[8]  P. Harremoës.  Information Topologies with Applications , 2007 .

[9]  Richard E. Blahut,et al.  Principles and practice of information theory , 1987 .

[10]  A. Antos,et al.  Convergence properties of functional estimates for discrete distributions , 2001 .

[11]  Gustav Herdan.  Principles of Information Theory , 1966 .

[12]  Robert Wieczorkowski,et al.  Entropy estimators‐improvements and comparisons , 1999 .

[13]  R. L. Dobrushin.  Passage to the Limit under the Information and Entropy Signs , 1960 .

[14]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[15]  P. Billingsley,et al.  Convergence of Probability Measures , 1970, The Mathematical Gazette.

[16]  A. Barron.  THE STRONG ERGODIC THEOREM FOR DENSITIES: GENERALIZED SHANNON-MCMILLAN-BREIMAN THEOREM' , 1985 .

[17]  Kellen Petersen August.  Real Analysis , 2009 .