Rényi Bounds on Information Combining

Bounds on information combining are entropic inequalities that determine how the information, or entropy, of a set of random variables can change when they are combined in certain prescribed ways. Such bounds play an important role in information theory, particularly in coding and Shannon theory. The arguably most elementary kind of information combining is the addition of two binary random variables, i.e. a CNOT gate, and the resulting quantities are fundamental when investigating belief propagation and polar coding.In this work we will generalize the concept to Rényi entropies. We give optimal bounds on the conditional Rényi entropy after combination, based on a certain convexity or concavity property and discuss when this property indeed holds. Since there is no generally agreed upon definition of the conditional Rényi entropy, we consider four different versions from the literature.Finally, we discuss the application of these bounds to the polarization of Rényi entropies under polar codes.

[1]  Varun Jog,et al.  The Entropy Power Inequality and Mrs. Gerber's Lemma for Abelian Groups of Order 2^n , 2012, ArXiv.

[2]  Cong Ling,et al.  On the Polarization of Rényi Entropy , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[3]  P. Jizba,et al.  The world according to R enyi: thermodynamics of multifractal systems , 2002, cond-mat/0207707.

[4]  W. Marsden.  I and J , 2012 .

[5]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[6]  Christian Cachin,et al.  Entropy measures and unconditional security in cryptography , 1997 .

[7]  Masahito Hayashi,et al.  Exponential Decreasing Rate of Leaked Information in Universal Random Privacy Amplification , 2009, IEEE Transactions on Information Theory.

[8]  Peter E. Latham,et al.  Mutual Information , 2006 .

[9]  Shahab Asoodeh,et al.  Generalizing Bottleneck Problems , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[10]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-II , 1973, IEEE Trans. Inf. Theory.

[11]  Junji Shikata,et al.  Information Theoretic Security for Encryption Based on Conditional Rényi Entropies , 2013, ICITS.

[12]  Berry Schoenmakers,et al.  Sharp lower bounds on the extractable randomness from non-uniform sources , 2011, Inf. Comput..

[13]  Imre Csiszár.  Generalized cutoff rates and Renyi's information measures , 1995, IEEE Trans. Inf. Theory.

[14]  Luis Filipe Coelho Antunes,et al.  Conditional Rényi Entropies , 2012, IEEE Transactions on Information Theory.

[15]  Abbas El Gamal,et al.  Network Information Theory , 2021, 2021 IEEE 3rd International Conference on Advanced Trends in Information Theory (ATIT).

[16]  Christoph Hirche,et al.  Bounds on Information Combining with Quantum Side Information , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[17]  Serge Fehr,et al.  On the Conditional Rényi Entropy , 2014, IEEE Transactions on Information Theory.

[18]  Saikat Guha,et al.  Polar Codes for Classical-Quantum Channels , 2011, IEEE Transactions on Information Theory.

[19]  David Burshtein,et al.  On the finite length scaling of ternary polar codes , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[20]  Jae Oh Woo,et al.  Majorization and Rényi entropy inequalities via Sperner theory , 2017, Discret. Math..

[21]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[22]  Emre Telatar,et al.  A simple proof of polarization and polarization for non-stationary channels , 2014, 2014 IEEE International Symposium on Information Theory.

[23]  Christoph Hirche,et al.  Polar codes in quantum information theory , 2015, ArXiv.

[24]  Marco Tomamichel,et al.  Quantum Information Processing with Finite Resources - Mathematical Foundations , 2015, ArXiv.

[25]  Jae Oh Woo,et al.  Entropy Inequalities for Sums in Prime Cyclic Groups , 2017, SIAM J. Discret. Math..

[26]  Venkatesan Guruswami,et al.  An Entropy Sumset Inequality and Polynomially Fast Convergence to Shannon Capacity Over All Alphabets , 2014, CCC.

[27]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-I , 1973, IEEE Trans. Inf. Theory.