Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions

We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the mutual information between this signal and its noisy version. The paper consists of essentially two parts. In the first, we derive several statistical-mechanical relationships between a few important quantities in this problem area, such as the MMSE, the differential entropy, the Fisher information, the free energy, and a generalized notion of temperature. We also draw analogies and differences between certain relations pertaining to the estimation problem and the parallel relations in thermodynamics and statistical physics. In the second part of the paper, we provide several application examples, where we demonstrate how certain analysis tools that are customary in statistical physics, prove useful in the analysis of the MMSE. In most of these examples, the corresponding statistical-mechanical systems turn out to consist of strong interactions that cause phase transitions, which in turn are reflected as irregularities and discontinuities (similar to threshold effects) in the behavior of the MMSE.

[1]  Neri Merhav.  The Generalized Random Energy Model and its Application to the Statistical Physics of Ensembles of Hierarchical Codes , 2009, IEEE Transactions on Information Theory.

[2]  Ido Kanter,et al.  Shannon meets Carnot: Generalized second thermodynamic law , 2009 .

[3]  Neri Merhav,et al.  Relations Between Random Coding Exponents and the Statistical Physics of Random Codes , 2007, IEEE Transactions on Information Theory.

[4]  E. Bolthausen,et al.  The Random Energy Model , 2002 .

[5]  B Derridai.  Magnetic properties and the function q ( x ) of the generalised random-energy model , .

[6]  F. Guerra.  Spin Glasses , 2005, cond-mat/0507581.

[7]  Daniel Pérez Palomar,et al.  Representation of Mutual Information Via Input Estimates , 2007, IEEE Transactions on Information Theory.

[8]  K. Narayanan,et al.  On the Thermodynamic Temperature of a General Distribution , 2007, 0711.1460.

[9]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[10]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[11]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[12]  R. Baxter.  Exactly solved models in statistical mechanics , 1982 .

[13]  西森 秀稔.  Statistical physics of spin glasses and information processing : an introduction , 2001 .

[14]  B. Derrida.  A generalization of the Random Energy Model which includes correlations between energies , 1985 .

[15]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[16]  H. Nishimori.  Statistical Physics of Spin Glasses and Information Processing , 2001 .

[17]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[18]  A. Bovier,et al.  Sharp asymptotics for metastability in the random field Curie-Weiss model , 2008, 0806.4478.

[19]  Josef Honerkamp,et al.  Statistical Physics: An Advanced Approach with Applications , 1998 .

[20]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[21]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[22]  B. Derrida.  Random-Energy Model: Limit of a Family of Disordered Models , 1980 .

[23]  Shlomo Shamai,et al.  Information Theory On extrinsic information of good binary codes operating over Gaussian channels , 2007, Eur. Trans. Telecommun..

[24]  Bernard Derrida,et al.  Magnetic properties and the function q(x) of the generalised random-energy model , 1986 .

[25]  N. Sourlas.  Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding , 1994 .

[26]  Toshiyuki TANAKA,et al.  Generic Multiuser Detection and Statistical Physics , 2009 .

[27]  Neri Merhav.  An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory , 2008, 2008 IEEE International Symposium on Information Theory.

[28]  Bernard Derrida,et al.  Solution of the generalised random energy model , 1986 .

[29]  B. Derrida.  Random-energy model: An exactly solvable model of disordered systems , 1981 .