An Entropy Sumset Inequality and Polynomially Fast Convergence to Shannon Capacity Over All Alphabets

We prove a lower estimate on the increase in entropy when two copies of a conditional random variable X|Y, with X supported on Zq = {0,1,..., q − 1} for prime q, are summed modulo q. Specifically, given two i.i.d. copies (X1, Y1) and (X2, Y2) of a pair of random variables (X, Y), with X taking values in Zq, we show H(X1 + X2 | Y1, Y2) - H(X|Y ) ≥ α(q) · H(X|Y)(1 - H(X|Y)) for some α(q) > 0, where H (·) is the normalized (by factor log2q) entropy. In particular, if X|Y is not close to being fully random or fully deterministic and H(X|Y) ∈ (γ, 1-γ), then the entropy of the sum increases by Ωq (γ). Our motivation is an effective analysis of the finite-length behavior of polar codes, for which the linear dependence on γ is quantitatively important. The assumption of q being prime is necessary: for X supported uniformly on a proper subgroup of Zq we have H(X + X) = H(X). For X supported on infinite groups without a finite subgroup (the torsion-free case) and no conditioning, a sumset inequality for the absolute increase in (unnormalized) entropy was shown by Tao in [20]. We use our sumset inequality to analyze Arikan's construction of polar codes and prove that for any q-ary source X, where q is any fixed prime, and any e > 0, polar codes allow efficient data compression of N i.i.d. copies of X into (H(X) + e)N q-ary symbols, as soon as N is polynomially large in 1/e. We can get capacity-achieving source codes with similar guarantees for composite alphabets, by factoring q into primes and combining different polar codes for each prime in factorization. A consequence of our result for noisy channel coding is that for all discrete memoryless channels, there are explicit codes enabling reliable communication within e > 0 of the symmetric Shannon capacity for a block length and decoding complexity bounded by a polynomial in 1/e. The result was previously shown for the special case of binary-input channels [7, 9], and this work extends the result to channels over any alphabet.

[1]  Varun Jog,et al.  The Entropy Power Inequality and Mrs. Gerber's Lemma for groups of order 2n , 2013, 2013 IEEE International Symposium on Information Theory.

[2]  Erdal Arikan,et al.  Source polarization , 2010, 2010 IEEE International Symposium on Information Theory.

[3]  R. Urbanke,et al.  Polar codes for Slepian-Wolf, Wyner-Ziv, and Gelfand-Pinsker , 2010, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[4]  Erdal Arikan,et al.  Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels , 2008, IEEE Transactions on Information Theory.

[5]  Emre Telatar,et al.  Polarization for arbitrary discrete memoryless channels , 2009, 2009 IEEE Information Theory Workshop.

[6]  Varun Jog,et al.  The Entropy Power Inequality and Mrs. Gerber's Lemma for Groups of Order 2n , 2014, IEEE Trans. Inf. Theory.

[7]  Rüdiger L. Urbanke,et al.  Polar codes: Characterization of exponent, bounds, and constructions , 2009, 2009 IEEE International Symposium on Information Theory.

[8]  Venkatesan Guruswami,et al.  Polar Codes: Speed of Polarization and Polynomial Gap to Capacity , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[9]  Emre Telatar,et al.  Polar Codes for the Two-User Multiple-Access Channel , 2010, IEEE Transactions on Information Theory.

[10]  Eren Sasoglu,et al.  Polarization and Polar Codes , 2012, Found. Trends Commun. Inf. Theory.

[11]  Hans S. Witsenhausen,et al.  Entropy inequalities for discrete channels , 1974, IEEE Trans. Inf. Theory.

[12]  Emre Telatar,et al.  On the rate of channel polarization , 2008, 2009 IEEE International Symposium on Information Theory.

[13]  Emre Telatar,et al.  A New Entropy Power Inequality for Integer-Valued Random Variables , 2013, IEEE Transactions on Information Theory.

[14]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-II , 1973, IEEE Trans. Inf. Theory.

[15]  Michael Gastpar,et al.  Polar Codes for Broadcast Channels , 2013, IEEE Transactions on Information Theory.

[16]  Eren Saso ˘ glu An entropy inequality for q-ary random variables and its application to channel polarization , 2010 .

[17]  Alexander Vardy,et al.  Achieving the secrecy capacity of wiretap channels using Polar codes , 2010, ISIT.

[18]  Emre Telatar,et al.  Polar Codes for the m-User MAC , 2010 .

[19]  Rüdiger L. Urbanke,et al.  Polar Codes are Optimal for Lossy Source Coding , 2009, IEEE Transactions on Information Theory.

[20]  Shai Shalev-Shwartz,et al.  Online Learning and Online Convex Optimization , 2012, Found. Trends Mach. Learn..

[21]  Emre Telatar,et al.  Polar Codes for the $m$-User Multiple Access Channel , 2012, IEEE Transactions on Information Theory.

[22]  Emmanuel Abbe,et al.  Polynomial complexity of polar codes for non-binary alphabets, key agreement and Slepian-Wolf coding , 2014, 2014 48th Annual Conference on Information Sciences and Systems (CISS).

[23]  Aaron D. Wyner,et al.  A theorem on the entropy of certain binary sequences and applications-I , 1973, IEEE Trans. Inf. Theory.

[24]  Lele Wang,et al.  Polar coding for interference networks , 2014, 2014 IEEE International Symposium on Information Theory.

[25]  Terence Tao,et al.  Sumset and Inverse Sumset Theory for Shannon Entropy , 2009, Combinatorics, Probability and Computing.

[26]  Eren Sasoglu An entropy inequality for q-ary random variables and its application to channel polarization , 2010, 2010 IEEE International Symposium on Information Theory.

[27]  Rüdiger L. Urbanke,et al.  Finite-Length Scaling for Polar Codes , 2013, IEEE Transactions on Information Theory.