Statistical Physics and Information Theory

1: Introduction 2: Basic Background in Statistical Physics 3: Physical Interpretations of Information Measures 4: Analysis Tools and Asymptotic Methods 5: Interacting Particles and Phase Transitions 6: The Random Energy Model and Random Coding 7: Extensions of the REM 8: Summary and Outlook, References.

[1]  D. Saad,et al.  Statistical mechanics of low-density parity-check codes , 2004 .

[2]  Tatsuto Murayama Statistical mechanics of the data compression theorem , 2002 .

[3]  R. Gallager Information Theory and Reliable Communication , 1968 .

[4]  C. Jarzynski Nonequilibrium Equality for Free Energy Differences , 1996, cond-mat/9610209.

[5]  Alexander Barg,et al.  Random codes: Minimum distances and error exponents , 2002, IEEE Trans. Inf. Theory.

[6]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[7]  Frederick Jelinek Tree encoding of memoryless time-discrete sources with a fidelity criterion , 1969, IEEE Trans. Inf. Theory.

[8]  C. G. Hoover,et al.  Nonequilibrium temperature and thermometry in heat-conducting phi4 models. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[9]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[10]  Howard Reiss,et al.  Hard spheres: Thermodynamics and geometry , 1989 .

[11]  Martin E. Hellman,et al.  On tree coding with a fidelity criterion , 1975, IEEE Trans. Inf. Theory.

[12]  Yoshiyuki Kabashima,et al.  Statistical Mechanical Approach to Error Exponents of Lossy Data Compression , 2003 .

[13]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[14]  Yoshiyuki Kabashima,et al.  How could the replica method improve accuracy of performance assessment of channel coding? , 2008, ArXiv.

[15]  Neri Merhav,et al.  On the Statistical Physics of Directed Polymers in a Random Medium and Their Relation to Tree Codes , 2008, IEEE Transactions on Information Theory.

[16]  J. Sethna Statistical Mechanics: Entropy, Order Parameters, and Complexity , 2021 .

[17]  西森 秀稔 Statistical physics of spin glasses and information processing : an introduction , 2001 .

[18]  E. Jaynes Information Theory and Statistical Mechanics , 1957 .

[19]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[20]  J. Parrondo,et al.  Dissipation: the phase-space perspective. , 2007, Physical review letters.

[21]  Rodney W. Johnson,et al.  Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy , 1980, IEEE Trans. Inf. Theory.

[22]  D. Saad,et al.  Statistical mechanics of error-correcting codes , 1999 .

[23]  S. Edwards,et al.  The Theory of Polymer Dynamics , 1986 .

[24]  Neri Merhav,et al.  Universal coding with minimum probability of codeword length overflow , 1991, IEEE Trans. Inf. Theory.

[25]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[26]  Frederick Jelinek,et al.  Buffer overflow in variable length coding of fixed rate sources , 1968, IEEE Trans. Inf. Theory.

[27]  G. Crooks Beyond Boltzmann-Gibbs statistics: maximum entropy hyperensembles out of equilibrium. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[28]  Rodger E. Ziemer,et al.  Principles of Communication , 2001 .

[29]  Wu-Sheng Dai,et al.  The explicit expression of the fugacity for hard-sphere Bose and Fermi gases , 2009 .

[30]  N. D. Bruijn Asymptotic methods in analysis , 1958 .

[31]  B. Derrida Random-Energy Model: Limit of a Family of Disordered Models , 1980 .

[32]  Thierry Mora,et al.  Statistical mechanics of error exponents for error-correcting codes , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[33]  Neri Merhav The Generalized Random Energy Model and its Application to the Statistical Physics of Ensembles of Hierarchical Codes , 2009, IEEE Transactions on Information Theory.

[34]  Gian Paolo Beretta,et al.  Rigorous and General Definition of Thermodynamic Entropy , 2011 .

[35]  Neri Merhav,et al.  Error exponents of optimum decoding for the degraded broadcast channel using moments of type class enumerators , 2008, 2009 IEEE International Symposium on Information Theory.

[36]  Howard Reiss Thermodynamic-like transformations in information theory , 1969 .

[37]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[38]  Neri Merhav Threshold Effects in Parameter Estimation as Phase Transitions in Statistical Mechanics , 2011, IEEE Trans. Inf. Theory.

[39]  Robert G. Gallager Tree encoding for symmetric sources with a distortion measure , 1974, IEEE Trans. Inf. Theory.

[40]  Neri Merhav,et al.  The random energy model in a magnetic field and joint source–channel coding , 2008, 0803.2789.

[41]  Henri Orland,et al.  White and weighted averages over solutions of Thouless Anderson Palmer equations for the Sherrington Kirkpatrick spin glass , 1980 .

[42]  Masato Okada,et al.  Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons , 2008, 2008 IEEE International Symposium on Information Theory.

[43]  Louis-Pierre Arguin Spin Glass Computations and Ruelle’s Probability Cascades , 2006 .

[44]  R. Gray Source Coding Theory , 1989 .

[45]  Charles H. Bennett,et al.  Notes on Landauer's Principle, Reversible Computation, and Maxwell's Demon , 2002, physics/0210005.

[46]  Yoshiyuki Kabashima,et al.  Statistical mechanics of source coding with a fidelity criterion , 2005 .

[47]  B. Derrida Random-energy model: An exactly solvable model of disordered systems , 1981 .

[48]  T. Werlang,et al.  Ergodic hypothesis in classical statistical mechanics , 2007 .

[49]  Masato Okada,et al.  Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons , 2008, 2008 IEEE International Symposium on Information Theory.

[50]  Raymond W. Yeung,et al.  A First Course in Information Theory , 2002 .

[51]  Daniel Pérez Palomar,et al.  Lautum Information , 2008, IEEE Transactions on Information Theory.

[52]  Pierre Maréchal,et al.  The role of perspective functions in convexity, polyconvexity, rank-one convexity and separate convexity , 2008 .

[53]  Frank Kelly,et al.  Reversibility and Stochastic Networks , 1979 .

[54]  T. D. Lee,et al.  Statistical Theory of Equations of State and Phase Transitions. II. Lattice Gas and Ising Model , 1952 .

[55]  Neri Merhav,et al.  Exact Random Coding Exponents for Erasure Decoding , 2011, IEEE Trans. Inf. Theory.

[56]  T. D. Lee,et al.  Statistical Theory of Equations of State and Phase Transitions. I. Theory of Condensation , 1952 .

[57]  A. H. W. Beck,et al.  Statistical mechanics, fluctuations and noise , 1976 .

[58]  Shlomo Shamai,et al.  Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions , 2008, IEEE Transactions on Information Theory.

[59]  Pierre A. Humblet Generalization of Huffman coding to minimize the probability of buffer overflow , 1981, IEEE Trans. Inf. Theory.

[60]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[61]  Mehran Kardar,et al.  Statistical physics of particles , 2007 .

[62]  Victor Dotsenko One more discussion of the replica trick: the examples of exact solutions , 2010 .

[63]  Neri Merhav Rate–Distortion Function via Minimum Mean Square Error Estimation , 2011, IEEE Transactions on Information Theory.

[64]  Andrew J. Viterbi,et al.  Principles of Digital Communication and Coding , 1979 .

[65]  Ruján Finite temperature error-correcting codes. , 1993, Physical review letters.

[66]  Neri Merhav,et al.  Relations Between Random Coding Exponents and the Statistical Physics of Random Codes , 2007, IEEE Transactions on Information Theory.

[67]  T. Teichmann,et al.  Elementary Statistical Physics , 1959 .

[68]  Josef Honerkamp,et al.  Statistical Physics: An Advanced Approach with Applications , 1998 .

[69]  John B. Anderson,et al.  Instrumentable tree encoding of information sources (Corresp.) , 1971, IEEE Trans. Inf. Theory.

[70]  Neri Merhav,et al.  Data Processing Theorems and the Second Law of Thermodynamics , 2010, IEEE Transactions on Information Theory.

[71]  Howard Reiss,et al.  Statistical Mechanics of Rigid Spheres , 1959 .

[72]  H Qian Relative entropy: free energy associated with equilibrium fluctuations and nonequilibrium deviations. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[73]  Neri Merhav,et al.  Exact Random Coding Exponents for Erasure Decoding , 2011, IEEE Transactions on Information Theory.

[74]  Neri Merhav On the physics of rate-distortion theory , 2010, 2010 IEEE International Symposium on Information Theory.

[75]  A. Wyner On the Probability of Buffer Overflow Under an Arbitrary Bounded Input-Output Distribution , 1974 .

[76]  B.D. Steinberg,et al.  Principles of communication , 1978, Proceedings of the IEEE.

[77]  L. Onsager Crystal statistics. I. A two-dimensional model with an order-disorder transition , 1944 .

[78]  Michael J. W. Hall,et al.  Universal geometric approach to uncertainty, entropy, and information , 1999 .

[79]  Bernard Derrida,et al.  Solution of the generalised random energy model , 1986 .

[80]  A. Montanari The glassy phase of Gallager codes , 2001, cond-mat/0104079.

[81]  A. Montanari Turbo codes: the phase transition , 2000, cond-mat/0003218.

[82]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[83]  Dov Levine,et al.  Nonequilibrium fluctuation theorems in the presence of local heating. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[84]  M. Kendall Elementary Statistics , 1945, Nature.

[85]  Giorgio Parisi,et al.  Mean field theory of the glass transition and jamming of hard spheres , 2008 .

[86]  Toshiyuki Tanaka,et al.  Statistical mechanics of CDMA multiuser demodulation , 2001 .

[87]  Neri Merhav Physics of the Shannon Limits , 2010, IEEE Transactions on Information Theory.

[88]  B. Scoppola,et al.  Statistical Mechanics Approach to Coding Theory , 1999 .

[89]  A. E. Patrick,et al.  Directed polymers on trees: a martingale approach , 1993 .

[90]  H. Löwen Fun with Hard Spheres , 2000 .

[91]  Neri Merhav,et al.  Error Exponents of Erasure/List Decoding Revisited Via Moments of Distance Enumerators , 2007, IEEE Transactions on Information Theory.

[92]  R. Landauer,et al.  Irreversibility and heat generation in the computing process , 1961, IBM J. Res. Dev..

[93]  I. M. Jacobs,et al.  Principles of Communication Engineering , 1965 .

[94]  E. Olivieri,et al.  Large deviations and metastability: Large deviations and statistical mechanics , 2005 .

[95]  G. B. Bagci The Physical Meaning of Renyi Relative Entropies , 2007 .

[96]  G. David Forney,et al.  Exponential error bounds for erasure, list, and decision feedback schemes , 1968, IEEE Trans. Inf. Theory.

[97]  Erik Ordentlich,et al.  Universal portfolios with side information , 1996, IEEE Trans. Inf. Theory.

[98]  Neri Merhav,et al.  Error exponents of optimum decoding for the interference channel , 2010, IEEE Trans. Inf. Theory.

[99]  E. Weinstein,et al.  Lower bounds on the mean square estimation error , 1985, Proceedings of the IEEE.

[100]  Bernard Derrida,et al.  The random energy model , 1980 .

[101]  H. Nishimori Statistical Physics of Spin Glasses and Information Processing , 2001 .

[102]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[103]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[104]  Thomas M. Cover,et al.  Elements of Information Theory: Cover/Elements of Information Theory, Second Edition , 2005 .

[105]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[106]  F. Reif,et al.  Fundamentals of Statistical and Thermal Physics , 1965 .

[107]  J. Ziv,et al.  A Generalization of the Rate-Distortion Theory and Applications , 1975 .

[108]  N. Sourlas Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding , 1994 .

[109]  Jacobus J. M. Verbaarschot,et al.  Critique of the replica trick , 1985 .

[110]  Howard Reiss,et al.  Statistical thermodynamic formalism in the solution of information theory problems , 1971 .

[111]  Jacob Ziv,et al.  On functionals satisfying a data-processing theorem , 1973, IEEE Trans. Inf. Theory.

[112]  Hugo Touchette,et al.  Methods for calculating nonconcave entropies , 2010, 1003.0382.

[113]  Toby Berger,et al.  Tree encoding of Gaussian sources , 1974, IEEE Trans. Inf. Theory.

[114]  R. Palmer,et al.  The replica method and solvable spin glass model , 1979 .

[115]  On the Thermodynamic Temperature of a General Distribution , 2007, 0711.1460.

[116]  R. Baxter Exactly solved models in statistical mechanics , 1982 .

[117]  Shlomo Shamai,et al.  Discrete-Input Two-Dimensional Gaussian Channels With Memory: Estimation and Information Rates Via Graphical Models and Statistical Mechanics , 2008, IEEE Transactions on Information Theory.

[118]  R. Ellis,et al.  Entropy, large deviations, and statistical mechanics , 1985 .

[119]  Nandor Simanyi,et al.  The Boltzmann--Sinai Ergodic Hypothesis In Full Generality , 2005, math/0510022.

[120]  G. D. J. Forney,et al.  On exponential error bounds for random codes on the BSC , 2001 .

[121]  R. Ellis,et al.  Large deviations and statistical mechanics , 1985 .

[122]  D. Ruelle Statistical Mechanics: Rigorous Results , 1999 .

[123]  Yoshiyuki Kabashima,et al.  Statistical mechanics of typical set decoding. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[124]  I. Csiszár A class of measures of informativity of observation channels , 1972 .

[125]  Neri Merhav Another Look at the Physics of Large Deviations With Application to Rate-Distortion Theory , 2009, ArXiv.

[126]  Ralf R. Müller,et al.  Replica analysis of general multiuser detection in MIMO DS-CDMA channels with imperfect CSI , 2008, 2008 IEEE International Symposium on Information Theory.

[127]  B. Derrida A generalization of the Random Energy Model which includes correlations between energies , 1985 .