Some Useful Integral Representations for Information-Theoretic Analyses

This work is an extension of our earlier article, where a well-known integral representation of the logarithmic function was explored and was accompanied with demonstrations of its usefulness in obtaining compact, easily-calculable, exact formulas for quantities that involve expectations of the logarithm of a positive random variable. Here, in the same spirit, we derive an exact integral representation (in one or two dimensions) of the moment of a nonnegative random variable, or the sum of such independent random variables, where the moment order is a general positive non-integer real (also known as fractional moments). The proposed formula is applied to a variety of examples with an information-theoretic motivation, and it is shown how it facilitates their numerical evaluations. In particular, when applied to the calculation of a moment of the sum of a large number, n, of nonnegative random variables, it is clear that integration over one or two dimensions, as suggested by our proposed integral representation, is significantly easier than the alternative of integrating over n dimensions, as needed in the direct calculation of the desired moment.

[1]  L. L. Campbell,et al.  A Coding Theorem and Rényi's Entropy , 1965, Inf. Control..

[2]  Dongfeng Yuan,et al.  Logarithmic Expectation of the Sum of Exponential Random Variables for Wireless Communication Performance Evaluation , 2015, 2015 IEEE 82nd Vehicular Technology Conference (VTC2015-Fall).

[3]  Newman,et al.  Interface growth and Burgers turbulence: The problem of random initial conditions. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[4]  Amos Lapidoth,et al.  Encoding Tasks and Rényi Entropy , 2014, IEEE Transactions on Information Theory.

[5]  S. Kak.  Information, physics, and computation , 1996 .

[6]  Rajesh Sundaresan,et al.  Guessing Under Source Uncertainty , 2006, IEEE Transactions on Information Theory.

[7]  Aldo Tagliani,et al.  Stieltjes moment problem via fractional moments , 2005, Appl. Math. Comput..

[8]  Neri Merhav,et al.  Joint Source-Channel Coding and Guessing with Application to Sequential Decoding , 1998, IEEE Trans. Inf. Theory.

[9]  The generalized Cauchy family of distributions with applications , 2016 .

[10]  Sergio Verdú,et al.  Variable-length lossy compression and channel coding: Non-asymptotic converses via cumulant generating functions , 2014, 2014 IEEE International Symposium on Information Theory.

[11]  C. Appledorn.  The Entropy of a Poisson Distribution , 1987 .

[12]  Rajesh Sundaresan,et al.  Guessing Revisited: A Large Deviations Approach , 2010, IEEE Transactions on Information Theory.

[13]  Igal Sason,et al.  Tight Bounds on the Rényi Entropy via Majorization with Applications to Guessing and Compression , 2018, Entropy.

[14]  Rajesh Sundaresan,et al.  DRDO – IISc Programme on Advanced Research in Mathematical Engineering Guessing Based On Length Functions ( TR-PME-2007-02 ) by , 2007 .

[15]  Serdar Boztas,et al.  Comments on 'An inequality on guessing and its application to sequential decoding' , 1997, IEEE Trans. Inf. Theory.

[16]  M. Marsili,et al.  Optimal work extraction and mutual information in a generalized Szilárd engine. , 2019, Physical review. E.

[17]  Neri Merhav,et al.  An Integral Representation of the Logarithmic Function with Applications in Information Theory , 2020, Entropy.

[18]  Igal Sason,et al.  Concentration of Measure Inequalities in Information Theory, Communications, and Coding , 2012, Found. Trends Commun. Inf. Theory.

[19]  Amos Lapidoth,et al.  Capacity bounds via duality with applications to multiple-antenna systems on flat-fading channels , 2003, IEEE Trans. Inf. Theory.

[20]  Sergio Verdú,et al.  Cumulant generating function of codeword lengths in optimal lossless compression , 2014, 2014 IEEE International Symposium on Information Theory.

[21]  Erdal Arikan.  An inequality on guessing and its application to sequential decoding , 1996, IEEE Trans. Inf. Theory.

[22]  Neri Merhav,et al.  Universal Randomized Guessing With Application to Asynchronous Decentralized Brute–Force Attacks , 2020, IEEE Transactions on Information Theory.

[23]  Sergio Verdú,et al.  Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures , 2018, IEEE Transactions on Information Theory.

[24]  M. Raginsky.  Concentration of Measure Inequalities in Information Theory, Communications, and Coding: Second Edition , 2014 .

[25]  Alfonso Martinez,et al.  Spectral efficiency of optical direct detection , 2007 .

[26]  Cihan Tepedelenlioglu,et al.  Stochastic Ordering of Fading Channels Through the Shannon Transform , 2013, IEEE Transactions on Information Theory.

[27]  Amos Lapidoth,et al.  Guessing Attacks on Distributed-Storage Systems , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[28]  Charles Knessl,et al.  Integral representations and asymptotic expansions for Shannon and Renyi entropies , 1998 .

[29]  Aldo Tagliani.  Hausdorff moment problem and fractional moments: A simplified procedure , 2011, Appl. Math. Comput..

[30]  Henryk Gzyl,et al.  Determination of the distribution of total loss from the fractional moments of its exponential , 2012, Appl. Math. Comput..

[31]  Neri Merhav.  Lower Bounds on Exponential Moments of the Quadratic Error in Parameter Estimation , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[32]  Victoria Kostina,et al.  A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications , 2017, Entropy.

[33]  Muriel Médard,et al.  Why Botnets Work: Distributed Brute-Force Attacks Need No Synchronization , 2018, IEEE Transactions on Information Forensics and Security.

[34]  Reshad Hosseini,et al.  Expected Logarithm of Central Quadratic Form and Its Use in KL-Divergence of Some Distributions , 2016, Entropy.

[35]  D. Berend,et al.  On the concentration of the missing mass , 2012, 1210.3248.

[36]  Kenneth E. Barner,et al.  A Generalized Cauchy Distribution Framework for Problems Requiring Robust Behavior , 2010, EURASIP J. Adv. Signal Process..

[37]  Aldo Tagliani,et al.  On the proximity of distributions in terms of coinciding fractional moments , 2003, Appl. Math. Comput..

[38]  G. Crooks.  On Measures of Entropy and Information , 2015 .

[39]  Neri Merhav,et al.  Guessing Subject to Distortion , 1998, IEEE Trans. Inf. Theory.