Statistical mechanics of error exponents for error-correcting codes

Error exponents characterize the exponential decay, when increasing message length, of the probability of error of many error-correcting codes. To tackle the long-standing problem of computing them exactly, we introduce a general, thermodynamic, formalism that we illustrate with maximum-likelihood decoding of low-density parity-check codes on the binary erasure channel and the binary symmetric channel. In this formalism, we apply the cavity method for large deviations to derive expressions for both the average and typical error exponents, which differ by the procedure used to select the codes from specified ensembles. When decreasing the noise intensity, we find that two phase transitions take place, at two different levels: a glass to ferromagnetic transition in the space of codewords and a paramagnetic to glass transition in the space of codes.

[1]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[2]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[3]  J. van Mourik,et al.  Average and reliability error exponents in low-density parity-check codes , 2003 .

[4]  Marc Mézard,et al.  Landscape of solutions in constraint satisfaction problems , 2005, Physical review letters.

[5]  Sergio Verdú,et al.  Fifty Years of Shannon Theory , 1998, IEEE Trans. Inf. Theory.

[6]  A. Montanari,et al.  On the Dynamics of the Glass Transition on Bethe Lattices , 2005, cond-mat/0509366.

[7]  Alexander Barg,et al.  Random codes: Minimum distances and error exponents , 2002, IEEE Trans. Inf. Theory.

[8]  M. Mézard,et al.  The Cavity Method at Zero Temperature , 2002, cond-mat/0207121.

[9]  T. Nattermann.  Theory of the Random Field Ising Model , 1997 .

[10]  Riccardo Zecchina,et al.  Alternative solutions to diluted p-spin models and XORSAT problems , 2002, ArXiv.

[11]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[12]  S. Franz,et al.  Dynamic phase transition for decoding algorithms. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[13]  Olivier Rivoire,et al.  Exactly solvable models of adaptive networks. , 2006, Physical review letters.

[14]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[15]  Bernie Mulgrew,et al.  Proceedings IEEE International Conference on Communications , 1989 .

[16]  Yuhong Yang,et al.  Information Theory, Inference, and Learning Algorithms , 2005 .

[17]  M. Mézard,et al.  Random multi-index matching problems , 2005, cond-mat/0507180.

[18]  Riccardo Zecchina,et al.  Simplest random K-satisfiability problem , 2000, Physical review. E, Statistical, nonlinear, and soft matter physics.

[19]  M. Mézard,et al.  Two Solutions to Diluted p-Spin Models and XORSAT Problems , 2003 .

[20]  Béla Bollobás,et al.  Random Graphs: Notation , 2001 .

[21]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.

[22]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[23]  A. Glavieux,et al.  Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1 , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[24]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[25]  J. van Mourik,et al.  The polynomial error probability for LDPC codes , 2003 .

[26]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[27]  D. Saad,et al.  Statistical mechanics of low-density parity-check codes , 2004 .

[28]  西森 秀稔.  Statistical physics of spin glasses and information processing : an introduction , 2001 .

[29]  Andrea Montanari,et al.  From Large Scale Rearrangements to Mode Coupling Phenomenology in Model Glasses , 2005 .

[30]  M. Mézard,et al.  The Bethe lattice spin glass revisited , 2000, cond-mat/0009418.

[31]  A. Montanari,et al.  On the nature of the low-temperature phase in discontinuous mean-field spin glasses , 2003, cond-mat/0301591.

[32]  A. Montanari.  The glassy phase of Gallager codes , 2001, cond-mat/0104079.

[33]  R. Monasson,et al.  Rigorous decimation-based construction of ground pure states for spin-glass models on random lattices. , 2002, Physical review letters.

[34]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[35]  Emre Telatar,et al.  Finite-length analysis of low-density parity-check codes on the binary erasure channel , 2002, IEEE Trans. Inf. Theory.

[36]  L. Goddard.  Information Theory , 1962, Nature.

[37]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[38]  Robert Michael Tanner,et al.  A recursive approach to low complexity codes , 1981, IEEE Trans. Inf. Theory.

[39]  Andrea Montanari,et al.  Weight distributions of LDPC code ensembles: combinatorics meets statistical physics , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[40]  Andrea Montanari,et al.  Life Above Threshold: From List Decoding to Area Theorem and MSE , 2004, ArXiv.

[41]  O. Rivoire.  The cavity method for large deviations , 2005, cond-mat/0506164.

[42]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[43]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[44]  G. Kiczales,et al.  Proceedings the , 1997 .

[45]  Thierry Mora,et al.  Error Exponents of Low-Density Parity-Check Codes on the Binary Erasure Channel , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.