The Generalized Random Energy Model and its Application to the Statistical Physics of Ensembles of Hierarchical Codes

In an earlier work, the statistical physics associated with finite-temperature decoding of code ensembles, along with the relation to their random coding error exponents, were explored in a framework that is analogous to Derrida's random energy model (REM) of spin glasses, according to which the energy levels of the various spin configurations are independent random variables. The generalized REM (GREM) extends the REM in that it introduces correlations between energy levels in an hierarchical structure. In this paper, we explore some analogies between the behavior of the GREM and that of code ensembles which have parallel hierarchical structures. In particular, in analogy to the fact that the GREM may have different types of phase transition effects, depending on the parameters of the model, then the above-mentioned hierarchical code ensembles behave substantially differently in the various domains of the design parameters of these codes. We make an attempt to explore the insights that can be imported from the statistical mechanics of the GREM and be harnessed to serve for code design considerations and guidelines.

[1]  Andrea Montanari,et al.  Two Lectures on Iterative Coding and Statistical Mechanics , 2005 .

[2]  T. C. Dorlas,et al.  PHASE DIAGRAM OF THE RANDOM ENERGY MODEL WITH HIGHER-ORDER FERROMAGNETIC TERM AND ERROR CORRECTING CODES DUE TO SOURLAS , 1999 .

[3]  Yoshiyuki Kabashima,et al.  Statistical Mechanical Approach to Error Exponents of Lossy Data Compression , 2003 .

[4]  Thierry Mora,et al.  Statistical mechanics of error exponents for error-correcting codes , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  Robert G. Gallager,et al.  The random coding bound is tight for the average code (Corresp.) , 1973, IEEE Trans. Inf. Theory.

[6]  Thomas M. Cover,et al.  Elements of information theory (2. ed.) , 2006 .

[7]  Bernard Derrida,et al.  Solution of the generalised random energy model , 1986 .

[8]  D. B. Saakian Diluted generalized random energy model , 1998 .

[9]  K. Adkins Theory of spin glasses , 1974 .

[10]  Andrew J. Viterbi,et al.  Principles of Digital Communication and Coding , 1979 .

[11]  D. Saad,et al.  Error-correcting codes that nearly saturate Shannon's bound , 1999, cond-mat/9906011.

[12]  Neri Merhav,et al.  Relations Between Random Coding Exponents and the Statistical Physics of Random Codes , 2007, IEEE Transactions on Information Theory.

[13]  Ruján Finite temperature error-correcting codes. , 1993, Physical review letters.

[14]  B. Derrida Random-Energy Model: Limit of a Family of Disordered Models , 1980 .

[15]  Aaron D. Wyner,et al.  The rate-distortion function for source coding with side information at the decoder , 1976, IEEE Trans. Inf. Theory.

[16]  Tatsuto Murayama Statistical mechanics of the data compression theorem , 2002 .

[17]  Andrea Montanari,et al.  Modern Coding Theory: The Statistical Mechanics and Computer Science Point of View , 2007, ArXiv.

[18]  N. Sourlas Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding , 1994 .

[19]  B. Derrida Random-energy model: An exactly solvable model of disordered systems , 1981 .

[20]  Yoshiyuki Kabashima,et al.  Statistical mechanics of source coding with a fidelity criterion , 2005 .

[21]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[22]  W. M. B. Dukes,et al.  On The Generalised Random Energy Model , 2001 .

[23]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[24]  Nicolas Sourlas,et al.  Statistical Mechanics and Error-Correcting Codes , 1990 .

[25]  Erik Ordentlich,et al.  Universal portfolios with side information , 1996, IEEE Trans. Inf. Theory.

[26]  Neri Merhav,et al.  Error Exponents of Erasure/List Decoding Revisited Via Moments of Distance Enumerators , 2007, IEEE Transactions on Information Theory.

[27]  Kohtaro Tadaki,et al.  A Statistical Mechanical Interpretation of Instantaneous Codes , 2007, 2007 IEEE International Symposium on Information Theory.

[28]  D. Saad,et al.  Tighter decoding reliability bound for Gallager's error-correcting code. , 2000, Physical review. E, Statistical, nonlinear, and soft matter physics.

[29]  Andrea Montanari,et al.  Tight bounds for LDPC and LDGM codes under MAP decoding , 2004, IEEE Transactions on Information Theory.

[30]  Anton Bovier,et al.  Derrida's Generalised Random Energy models 1: models with finitely many hierarchies , 2004 .

[31]  Bixio Rimoldi,et al.  Successive refinement of information: characterization of the achievable rates , 1994, IEEE Trans. Inf. Theory.

[32]  A. D. Wyner,et al.  The wire-tap channel , 1975, The Bell System Technical Journal.

[33]  S. Franz,et al.  Dynamic phase transition for decoding algorithms. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  Bernard Derrida,et al.  Magnetic properties and the function q(x) of the generalised random-energy model , 1986 .

[35]  William Equitz,et al.  Successive refinement of information , 1991, IEEE Trans. Inf. Theory.

[36]  B. Derrida A generalization of the Random Energy Model which includes correlations between energies , 1985 .

[37]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[38]  J. Nadal,et al.  From statistical physics to statistical inference and back , 1994 .

[39]  Bernard Derrida,et al.  The probability distribution of the partition function of the random energy model , 1989 .

[40]  Giorgio Parisi,et al.  Crossover behavior of a one-dimensional random energy model , 1998 .

[41]  Nabin Kumar Jana Exponential Random Energy Model , 2006 .

[42]  E. Bolthausen,et al.  The Random Energy Model , 2002 .

[43]  B Derridai Magnetic properties and the function q ( x ) of the generalised random-energy model , .

[44]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[45]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[46]  D. Saad,et al.  Statistical mechanics of error-correcting codes , 1999 .

[47]  A. Montanari The glassy phase of Gallager codes , 2001, cond-mat/0104079.

[48]  B. Scoppola,et al.  Statistical Mechanics Approach to Coding Theory , 1999 .