Analysis of Mismatched Estimation Errors Using Gradients of Partition Functions

We consider the problem of signal estimation (denoising) from a statistical-mechanical perspective, in continuation to a recent work on the analysis of mean-square error (MSE) estimation using a direct relationship between optimum estimation and certain partition functions. This paper consists of essentially two parts. In the first part, using the aforementioned relationship, we derive single-letter expressions of the asymptotic mismatched MSE of a codeword (from a randomly selected code), corrupted by a Gaussian vector channel. In the second part, we provide several examples to demonstrate phase transitions in the behavior of the MSE. These examples enable us to understand more deeply and to gather intuition regarding the roles of the real and the mismatched probability measures in creating these phase transitions.

[1]  O. Frink.  Differentiation of sequences , 1935 .

[2]  Robert M. Gray,et al.  Toeplitz And Circulant Matrices: A Review (Foundations and Trends(R) in Communications and Information Theory) , 2006 .

[3]  Tsachy Weissman,et al.  Universal Minimax Discrete Denoising Under Channel Uncertainty , 2006, IEEE Transactions on Information Theory.

[4]  Y. Iba.  The Nishimori line and Bayesian statistics , 1998, cond-mat/9809190.

[5]  T. Kailath.  The innovations approach to detection and estimation theory , 1970 .

[6]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[7]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[8]  Shlomo Shamai,et al.  Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions , 2008, IEEE Transactions on Information Theory.

[9]  van Aernout Enter.  Statistical Mechanics, A Short Treatise , 2000 .

[10]  Sergio Verdú,et al.  Mismatched Estimation and Relative Entropy , 2009, IEEE Transactions on Information Theory.

[11]  Albrecht Böttcher,et al.  Spectral properties of banded Toeplitz matrices , 1987 .

[12]  S. Kak.  Information, physics, and computation , 1996 .

[13]  N. Sourlas.  Spin Glasses, Error-Correcting Codes and Finite-Temperature Decoding , 1994 .

[14]  Y. Kabashima,et al.  Statistical Mechanical Approach to Error Exponents of Lossy Data Compression , 2003, cond-mat/0311123.

[15]  Dongning Guo,et al.  Relative entropy and score function: New information-estimation relationships through arbitrary additive perturbation , 2009, 2009 IEEE International Symposium on Information Theory.

[16]  Daniel Pérez Palomar,et al.  Gradient of mutual information in linear vector Gaussian channels , 2006, IEEE Transactions on Information Theory.

[17]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[18]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[19]  Yoshiyuki Kabashima,et al.  Statistical mechanics of source coding with a fidelity criterion , 2005 .

[20]  V. Akila,et al.  Information , 2001, The Lancet.

[21]  C. Shannon.  Probability of error for optimal codes in a Gaussian channel , 1959 .

[22]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[23]  Giovanni Gallavotti,et al.  Statistical Mechanics: A Short Treatise , 1999 .

[24]  Shlomo Shamai,et al.  Mutual Information and Conditional Mean Estimation in Poisson Channels , 2004, IEEE Transactions on Information Theory.

[25]  Robert M. Gray,et al.  Toeplitz and Circulant Matrices: A Review , 2005, Found. Trends Commun. Inf. Theory.

[26]  T. Duncan.  ON THE CALCULATION OF MUTUAL INFORMATION , 1970 .

[27]  Jonathan R. Partington,et al.  Spectral properties of banded Toeplitz matrices , 2007 .

[28]  W. Rudin.  Principles of mathematical analysis , 1964 .

[29]  Richard S. Bucy,et al.  Information and filtering , 1979, Inf. Sci..

[30]  Nicolas Sourlas,et al.  Spin-glass models as error-correcting codes , 1989, Nature.

[31]  Yoshiyuki Kabashima,et al.  Statistical mechanics of typical set decoding. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[32]  Todd P. Coleman,et al.  Mutual information and posterior estimates in channels of exponential family type , 2009, 2009 IEEE Information Theory Workshop.

[33]  D. Donoho.  ON MINIMUM ENTROPY DECONVOLUTION , 1981 .

[34]  Masato Okada,et al.  Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons , 2008, 2008 IEEE International Symposium on Information Theory.

[35]  Tsachy Weissman,et al.  Universal minimax discrete denoising under channel uncertainty , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[36]  Tsachy Weissman,et al.  The Relationship Between Causal and Noncausal Mismatched Estimation in Continuous-Time AWGN Channels , 2010, IEEE Transactions on Information Theory.

[37]  Majid Fozunbal,et al.  On regret of parametric mismatch in minimum mean square error estimation , 2010, 2010 IEEE International Symposium on Information Theory.

[38]  A. D. Wyner,et al.  A Bound on the Number of Distinguishable Functions which are Time-Limited and Approximately Band-Limited , 1973 .

[39]  Emre Telatar,et al.  Mismatched decoding revisited: General alphabets, channels with memory, and the wide-band limit , 2000, IEEE Trans. Inf. Theory.

[40]  Isidore Isaac Hirschman,et al.  Studies in real and complex analysis , 1965 .

[41]  Ismail Nikoufar,et al.  Mathematical Analysis II , 2016 .

[42]  Shlomo Shamai,et al.  Additive non-Gaussian noise channels: mutual information and conditional mean estimation , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[43]  Universal discrete denoising: known channel , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..

[44]  Daniel Pérez Palomar,et al.  Representation of Mutual Information Via Input Estimates , 2007, IEEE Transactions on Information Theory.

[45]  Toshiyuki Tanaka,et al.  Optimal spreading sequences in large CDMA systems: A statistical mechanics approach , 2008, 2008 IEEE International Symposium on Information Theory.

[46]  U. Grenander,et al.  Toeplitz Forms And Their Applications , 1958 .

[47]  T. Weissman.  The relationship between causal and non-causal mismatched estimation in continuous-time AWGN channels , 2010, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[48]  Tsachy Weissman,et al.  Mutual information, relative entropy, and estimation in the Poisson channel , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[49]  Shlomo Shamai,et al.  MMSE of “Bad” Codes , 2013, IEEE Transactions on Information Theory.

[50]  Tsachy Weissman,et al.  Denoising via MCMC-Based Lossy Compression , 2012, IEEE Transactions on Signal Processing.

[51]  Andrea Montanari,et al.  Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising , 2011, IEEE Transactions on Information Theory.

[52]  Jerzy Seidler,et al.  Bounds on the mean-square error and the quality of domain decisions based on mutual information , 1971, IEEE Trans. Inf. Theory.

[53]  Joan Jacobs,et al.  Analysis of Mismatched Estimation Errors Using Gradients of Partition Functions , 2013 .

[54]  Neri Merhav.  Optimum Estimation via Gradients of Partition Functions and Information Measures: A Statistical-Mechanical Perspective , 2011, IEEE Transactions on Information Theory.