Sparse Superposition Codes: Fast and Reliable at Rates Approaching Capacity with Gaussian Noise

For the additive white Gaussian noise channel with average codeword power constraint, sparse superposition codes are developed. Both encoding and decoding are computationally feasible. The codewords are linear combinations of subsets of vectors from a given dictionary, with the possible messages indexed by the choice of subset. An adaptive successive decoder is developed, with which communication is shown to be reliable with error probability exponentially small for all rates below the Shannon capacity.

[1]  William Feller,et al.  An Introduction to Probability Theory and Its Applications , 1951 .

[2]  J. Wolfowitz.  Review: William Feller, An introduction to probability theory and its applications. Vol. I , 1951 .

[3]  P. Whittle,et al.  Bounds for the Moments of Linear and Quadratic Forms in Independent Variables , 1960 .

[4]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[5]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[6]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[7]  Thomas M. Cover,et al.  Broadcast channels , 1972, IEEE Trans. Inf. Theory.

[8]  F. MacWilliams,et al.  The Theory of Error-Correcting Codes , 1977 .

[9]  I. Csiszár.  Sanov Property, Generalized $I$-Projection and a Conditional Limit Theorem , 1984 .

[10]  W. Fischer,et al.  Sphere Packings, Lattices and Groups , 1990 .

[11]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[12]  L. Jones.  A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training , 1992 .

[13]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[14]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[15]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[16]  Daniel A. Spielman,et al.  Expander codes , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[17]  D. Vernon.  Inform , 1995, Encyclopedia of the UN Sustainable Development Goals.

[18]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[19]  Rüdiger L. Urbanke,et al.  A rate-splitting approach to the Gaussian multiple-access channel , 1996, IEEE Trans. Inf. Theory.

[20]  Daniel J. Costello,et al.  A distance spectrum interpretation of turbo codes , 1996, IEEE Trans. Inf. Theory.

[21]  Joachim Hagenauer,et al.  Iterative decoding of binary block and convolutional codes , 1996, IEEE Trans. Inf. Theory.

[22]  Thomas C. Hales.  Sphere packings, I , 1997, Discret. Comput. Geom..

[23]  G. David Forney,et al.  Modulation and Coding for Linear Gaussian Channels , 1998, IEEE Trans. Inf. Theory.

[24]  Daniel A. Spielman,et al.  Efficient erasure correcting codes , 2001, IEEE Trans. Inf. Theory.

[25]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[26]  Daniel A. Spielman,et al.  Improved low-density parity-check codes using irregular graphs and belief propagation , 1998, Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252).

[27]  Shu Lin,et al.  Error Control Coding , 2004 .

[28]  A. Barg,et al.  Error Exponents of Expander Codes under Linear-Complexity Decoding , 2004, SIAM J. Discret. Math..

[29]  Jean-Jacques Fuchs,et al.  Recovery of exact sparse representations in the presence of bounded noise , 2005, IEEE Transactions on Information Theory.

[30]  Mohit Garg.  Multi-user Detection , 2005 .

[31]  Joel A. Tropp,et al.  Applications of sparse approximation in communications , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[32]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[33]  Michael Lentmaier,et al.  Joint Permutor Analysis and Design for Multiple Turbo Codes , 2006, IEEE Transactions on Information Theory.

[34]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[35]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[36]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[37]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[38]  Daniel J. Costello,et al.  Channel coding: The road to channel capacity , 2006, Proceedings of the IEEE.

[39]  Jian Cao,et al.  Asymptotically Optimal Multiple-Access Communication Via Distributed Rate Splitting , 2006, IEEE Transactions on Information Theory.

[40]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[41]  J. Tropp.  Norms of Random Submatrices and Sparse Approximation , 2008 .

[42]  A. Barron,et al.  Approximation and learning by greedy algorithms , 2008, 0803.1718.

[43]  J. Tropp.  On the conditioning of random subdictionaries , 2008 .

[44]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[45]  Sundeep Rangan,et al.  Necessary and Sufficient Conditions for Sparsity Pattern Recovery , 2008, IEEE Transactions on Information Theory.

[46]  Emre Telatar,et al.  On the rate of channel polarization , 2008, 2009 IEEE International Symposium on Information Theory.

[47]  Martin J. Wainwright,et al.  Information-Theoretic Limits on Sparsity Recovery in the High-Dimensional and Noisy Setting , 2007, IEEE Transactions on Information Theory.

[48]  E. Candès,et al.  Near-ideal model selection by ℓ1 minimization , 2008, 0801.0345.

[49]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[50]  G. David Forney,et al.  Concatenated codes , 2009, Scholarpedia.

[51]  Andrew R. Barron,et al.  Toward fast reliable communication at rates near capacity with Gaussian noise , 2010, 2010 IEEE International Symposium on Information Theory.

[52]  H. Vincent Poor,et al.  Channel Coding Rate in the Finite Blocklength Regime , 2010, IEEE Transactions on Information Theory.

[53]  Aaron B. Wagner,et al.  Moderate deviation analysis of channel coding: Discrete memoryless case , 2010, 2010 IEEE International Symposium on Information Theory.

[54]  Jianqing Fan,et al.  A Selective Overview of Variable Selection in High Dimensional Feature Space. , 2009, Statistica Sinica.

[55]  S. Verdú,et al.  The impact of constellation cardinality on Gaussian channel capacity , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[56]  Emre Telatar,et al.  Polar Codes for the m-User MAC , 2010 .

[57]  Matthew Malloy,et al.  Sequential analysis in high-dimensional multiple testing and sparse recovery , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[58]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[59]  Tong Zhang.  Multi-stage Convex Relaxation for Feature Selection , 2011, 1106.0565.

[60]  Tong Zhang,et al.  Sparse Recovery With Orthogonal Matching Pursuit Under RIP , 2010, IEEE Transactions on Information Theory.

[61]  Tong Zhang,et al.  Adaptive Forward-Backward Greedy Algorithm for Learning Sparse Representations , 2011, IEEE Transactions on Information Theory.

[62]  Pedram Pad,et al.  Capacity Achieving Random Sparse Linear Codes , 2011, ArXiv.

[63]  A. Sridharan.  Broadcast Channels , 2022 .