An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory

An identity between two versions of the large deviations rate function of the probability a certain rare event is established. This identity has an interpretation in statistical physics, namely, an isothermal equilibrium of a composite system that consists of multiple subsystems. Several information-theoretic application examples, where the analysis of this large deviations probability naturally arises, are then described from the viewpoint of this statistical mechanical interpretation. This results in several relationships between information theory and statistical physics, which we hope, the reader will find insightful.

[1]  David L. Neuhoff,et al.  Quantization , 2022, IEEE Trans. Inf. Theory.

[2]  M. B. Plenio,et al.  The physics of forgetting: Landauer's erasure principle and information theory , 2001, quant-ph/0103108.

[3]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[4]  Imre Csiszár,et al.  Channel capacity for a given decoding metric , 1995, IEEE Trans. Inf. Theory.

[5]  Shlomo Shamai,et al.  On information rates for mismatched decoders , 1994, IEEE Trans. Inf. Theory.

[6]  E. Jaynes.  On the rationale of maximum-entropy methods , 1982, Proceedings of the IEEE.

[7]  Amos Lapidoth,et al.  Mismatched decoding and the multiple-access channel , 1994, IEEE Trans. Inf. Theory.

[8]  Tsachy Weissman,et al.  Denoising and Filtering Under the Probability of Excess Loss Criterion , 2007, IEEE Transactions on Information Theory.

[9]  J. Cadzow.  Maximum Entropy Spectral Analysis , 2006 .

[10]  Zevi W. Salsburg,et al.  Elementary statistical physics , 1959 .

[11]  G. Longo.  Source Coding Theory , 1970 .

[12]  G. B. Bagci.  The Physical Meaning of Renyi Relative Entropies , 2007 .

[13]  Toshiyuki Tanaka,et al.  Statistical mechanics of CDMA multiuser demodulation , 2001 .

[14]  Yoshiyuki Kabashima,et al.  Statistical Mechanical Approach to Error Exponents of Lossy Data Compression , 2003 .

[15]  Yoshitsugu Oono,et al.  Large Deviation and Statistical Physics , 1989 .

[16]  Hidetoshi Nishimori.  Spin glasses and information , 2007 .

[17]  D. Saad,et al.  Error-correcting codes that nearly saturate Shannon's bound , 1999, cond-mat/9906011.

[18]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[19]  Tatsuto Murayama.  Statistical mechanics of the data compression theorem , 2002 .

[20]  David McAllester.  A Statistical Mechanics Approach to Large Deviation Theorems , 2007 .

[21]  O. J. E. Maroney.  The (absence of a) relationship between thermodynamic and logical reversibility , 2004 .

[22]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[23]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[24]  Thomas M. Cover,et al.  Elements of Information Theory: Cover/Elements of Information Theory, Second Edition , 2005 .

[25]  Shlomo Shamai,et al.  A lower bound on the bit-error-rate resulting from mismatched viterbi decoding , 1998, Eur. Trans. Telecommun..

[26]  R. Ellis,et al.  Entropy, large deviations, and statistical mechanics , 1985 .

[27]  Thierry Mora,et al.  Statistical mechanics of error exponents for error-correcting codes , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[28]  Neri Merhav,et al.  Source coding exponents for zero-delay coding with finite memory , 2003, IEEE Trans. Inf. Theory.

[29]  Robert M. Gray,et al.  Rate-distortion speech coding with a minimum discrimination information distortion measure , 1981, IEEE Trans. Inf. Theory.

[30]  R. Landauer,et al.  Irreversibility and heat generation in the computing process , 1961, IBM J. Res. Dev..

[31]  I Rojdestvenski,et al.  Mapping of statistical physics to information theory with application to biological systems. , 2000, Journal of theoretical biology.

[32]  Richard S. Ellis.  An overview of the theory of large deviations and applications to statistical mechanics. , 1996 .

[33]  David L. Neuhoff,et al.  Causal source codes , 1982, IEEE Trans. Inf. Theory.

[34]  R. Ellis,et al.  Large deviations and statistical mechanics , 1985 .

[35]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[36]  Imre Csiszár.  Generalized cutoff rates and Renyi's information measures , 1995, IEEE Trans. Inf. Theory.

[37]  N. Sourlas.  Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding , 1994 .

[38]  D. Saad,et al.  Tighter decoding reliability bound for Gallager's error-correcting code. , 2000, Physical review. E, Statistical, nonlinear, and soft matter physics.

[39]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[40]  Vladimir B. Balakirsky.  A converse coding theorem for mismatched decoding at the output of binary-input memoryless channels , 1995, IEEE Trans. Inf. Theory.

[41]  D. Saad,et al.  Statistical mechanics of error-correcting codes , 1999 .

[42]  B. Scoppola,et al.  Statistical Mechanics Approach to Coding Theory , 1999 .

[43]  Yoshiyuki Kabashima,et al.  Statistical mechanics of source coding with a fidelity criterion , 2005 .

[44]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[45]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[46]  Yoshiyuki Kabashima,et al.  Statistical mechanics of typical set decoding. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[47]  Y. Iba.  The Nishimori line and Bayesian statistics , 1998, cond-mat/9809190.

[48]  Rodney W. Johnson,et al.  Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy , 1980, IEEE Trans. Inf. Theory.

[49]  Stephen P. Boyd,et al.  Geometric programming duals of channel capacity and rate distortion , 2004, IEEE Transactions on Information Theory.