Another Look at the Physics of Large Deviations With Application to Rate-Distortion Theory

We revisit and extend the physical interpretation recently given to a certain identity between large--deviations rate--functions (as well as applications of this identity to Information Theory), as an instance of thermal equilibrium between several physical systems that are brought into contact. Our new interpretation, of mechanical equilibrium between these systems, is shown to have several advantages relative to that of thermal equilibrium. This physical point of view also provides a trigger to the development of certain alternative representations of the rate--distortion function and channel capacity, which are new to the best knowledge of the author.

[1]  Kenneth Rose,et al.  A mapping approach to rate-distortion computation and analysis , 1994, IEEE Trans. Inf. Theory.

[2]  David McAllester A Statistical Mechanics Approach to Large Deviation Theorems , 2007 .

[3]  H Qian Relative entropy: free energy associated with equilibrium fluctuations and nonequilibrium deviations. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[4]  Neri Merhav An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory , 2008, 2008 IEEE International Symposium on Information Theory.

[5]  E. M.,et al.  Statistical Mechanics , 2021, On Complementarity.

[6]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[7]  S. Kak Information, physics, and computation , 1996 .

[8]  R. J. Hawkins Statistical Mechanics, Fluctuations and Noise , 1977 .

[9]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[10]  G. B. Bagci Some remarks on R\'{e}nyi relative entropy in a thermostatistical framework , 2007 .

[11]  G. B. Bagci The Physical Meaning of Renyi Relative Entropies , 2007 .

[12]  Ido Kanter,et al.  Shannon meets Carnot: Generalized second thermodynamic law , 2009 .

[13]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[14]  G. Longo Source Coding Theory , 1970 .

[15]  E. Weinstein,et al.  Lower bounds on the mean square estimation error , 1985, Proceedings of the IEEE.