Logarithmic Sobolev inequalities and strong data processing theorems for discrete channels

The noisiness of a channel can be measured by comparing suitable functionals of the input and output distributions. For instance, if we fix a reference input distribution, then the worst-case ratio of output relative entropy to input relative entropy for any other input distribution is bounded by one, by the data processing theorem. However, for a fixed reference input distribution, this quantity may be strictly smaller than one, giving so-called strong data processing inequalities (SDPIs). This paper shows that the problem of determining both the best constant in an SDPI and any input distributions that achieve it can be addressed using so-called logarithmic Sobolev inequalities, which relate input relative entropy to certain measures of input-output correlation. Another contribution is a proof of equivalence between SDPIs and a limiting case of certain strong data processing inequalities for the Rényi divergence.

[1]  S. Bobkov,et al.  Modified Logarithmic Sobolev Inequalities in Discrete Settings , 2006 .

[2]  Venkat Anantharam,et al.  Non-interactive simulation of joint distributions: The Hirschfeld-Gebelein-Rényi maximal correlation and the hypercontractivity ribbon , 2012, 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[3]  Venkat Anantharam,et al.  On Maximal Correlation, Hypercontractivity, and the Data Processing Inequality studied by Erkip and Cover , 2013, ArXiv.

[4]  P. Gács,et al.  Spreading of Sets in Product Spaces and Hypercontraction of the Markov Operator , 1976 .

[5]  P. Diaconis,et al.  LOGARITHMIC SOBOLEV INEQUALITIES FOR FINITE MARKOV CHAINS , 1996 .

[6]  Peter Harremoës,et al.  Rényi Divergence and Kullback-Leibler Divergence , 2012, IEEE Transactions on Information Theory.

[7]  Elchanan Mossel,et al.  On Reverse Hypercontractivity , 2011, Geometric and Functional Analysis.

[8]  H. Witsenhausen.  ON SEQUENCES OF PAIRS OF DEPENDENT RANDOM VARIABLES , 1975 .

[9]  G. Crooks.  On Measures of Entropy and Information , 2015 .

[10]  L. Miclo,et al.  Remarques sur l’hypercontractivité et l’évolution de l’entropie pour des chaînes de Markov finies , 1997 .

[11]  P. Diaconis,et al.  Stochastic Alternating Projections , 2010 .

[12]  Igor Vajda,et al.  On Divergences and Informations in Statistics and Information Theory , 2006, IEEE Transactions on Information Theory.

[13]  D. Bakry.  L'hypercontractivité et son utilisation en théorie des semigroupes , 1994 .