Discrete-Input Two-Dimensional Gaussian Channels With Memory: Estimation and Information Rates Via Graphical Models and Statistical Mechanics

Discrete-input two-dimensional (2D) Gaussian channels with memory represent an important class of systems, which appears extensively in communications and storage. In spite of their widespread use, the workings of 2D channels are still very much unknown. In this work, we try to explore their properties from the perspective of estimation theory and information theory. At the heart of our approach is a mapping of a 2D channel to an undirected graphical model, and inferring its a posteriori probabilities (APPs) using generalized belief propagation (GBP). The derived probabilities are shown to be practically accurate, thus enabling optimal maximum a posteriori (MAP) estimation of the transmitted symbols. Also, the Shannon-theoretic information rates are deduced either via the vector-wise Shannon-McMillan-Breiman (SMB) theorem, or via the recently derived symbol-wise Guo-Shamai-Verdu (GSV) theorem. Our approach is also described from the perspective of statistical mechanics, as the graphical model and inference algorithm have their analogues in physics. Our experimental study, based on common channel settings taken from cellular networks and magnetic recording devices, demonstrates that under nontrivial memory conditions, the performance of this fully tractable GBP estimator is almost identical to the performance of the optimal MAP estimator. It also enables a practically accurate simulation-based estimate of the information rate. Rationalization of this excellent performance of GBP in the 2-D Gaussian channel setting is addressed.

[1]  Joseph A. O'Sullivan,et al.  Iterative decoding and equalization for 2-D recording channels , 2002 .

[2]  Alain Glavieux,et al.  Reflections on the Prize Paper : "Near optimum error-correcting coding and decoding: turbo codes" , 1998 .

[3]  Valerii M. Vinokur,et al.  Vortices in high-temperature superconductors , 1994 .

[4]  J. Laurie Snell,et al.  Markov Random Fields and Their Applications , 1980 .

[5]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[6]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[7]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[8]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[9]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[10]  R. Baxter Exactly solved models in statistical mechanics , 1982 .

[11]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Aleksandar Kavcic,et al.  Markov sources achieve the feedback capacity of finite-state machine channels , 2002, Proceedings IEEE International Symposium on Information Theory,.

[13]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[14]  John Cocke,et al.  Optimal decoding of linear codes for minimizing symbol error rate (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[15]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[16]  E. Lindenstrauss Pointwise theorems for amenable groups , 1999 .

[17]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[18]  P. Gehler,et al.  An introduction to graphical models , 2001 .

[19]  Shlomo Shamai,et al.  On the achievable information rates of finite-state input two-dimensional channels with memory , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[20]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[21]  P. Siegel,et al.  Information rates of two-dimensional finite state ISI channels , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..

[22]  Benjamin Weiss,et al.  Entropy and recurrence rates for stationary random fields , 2002, IEEE Trans. Inf. Theory.

[23]  Shlomo Shamai,et al.  The intersymbol interference channel: lower bounds on capacity and channel precoding loss , 1996, IEEE Trans. Inf. Theory.

[24]  Bane Vasic,et al.  Coding and Signal Processing for Magnetic Recording Systems , 2004 .

[25]  Andries P. Hekstra,et al.  Signal processing and coding for two-dimensional optical storage , 2003, GLOBECOM '03. IEEE Global Telecommunications Conference (IEEE Cat. No.03CH37489).

[26]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[27]  Michael I. Jordan Learning in Graphical Models , 1999, NATO ASI Series.

[28]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[29]  Jack K. Wolf,et al.  Iterative detection of 2-dimensional ISI channels , 2003, Proceedings 2003 IEEE Information Theory Workshop (Cat. No.03EX674).

[30]  Brendan J. Frey,et al.  Graphical Models for Machine Learning and Digital Communication , 1998 .

[31]  J.E. Mazo,et al.  Digital communications , 1985, Proceedings of the IEEE.

[32]  Invited,et al.  Abstract , 1985 .

[33]  Aaron D. Wyner,et al.  Shannon-theoretic approach to a Gaussian cellular multiple-access channel , 1994, IEEE Trans. Inf. Theory.

[34]  Joseph A. O'Sullivan,et al.  Joint equalization and decoding for nonlinear two-dimensional intersymbol interference channels , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[35]  M.A. Iqbal,et al.  Simulation-based estimation of the capacity of full-surface channels , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[36]  Keith M. Chugg,et al.  Two-dimensional equalization in coherent and incoherent page-oriented optical memory , 1999 .

[37]  Paul H. Siegel,et al.  Markov Processes Asymptotically Achieve the Capacity of Finite-State Intersymbol Interference Channels , 2004, IEEE Transactions on Information Theory.

[38]  Aleksandar Kavcic On the capacity of Markov sources over noisy channels , 2001, GLOBECOM'01. IEEE Global Telecommunications Conference (Cat. No.01CH37270).

[39]  Sergio Verdu,et al.  Multiuser Detection , 1998 .

[40]  Wei Zeng,et al.  Simulation-Based Computation of Information Rates for Channels With Memory , 2006, IEEE Transactions on Information Theory.

[41]  Walter Hirt Capacity and information rates of discrete-time channels with memory , 1988 .

[42]  Shlomo Shamai,et al.  Information rates for a discrete-time Gaussian channel with intersymbol interference and stationary inputs , 1991, IEEE Trans. Inf. Theory.

[43]  L. Onsager Crystal statistics. I. A two-dimensional model with an order-disorder transition , 1944 .

[44]  Paul H. Siegel,et al.  On the symmetric information rate of two-dimensional finite-state ISI channels , 2006, IEEE Transactions on Information Theory.

[45]  H. Nishimori Statistical Physics of Spin Glasses and Information Processing , 2001 .

[46]  William Weeks Full-Surface Data Storage , 2000 .

[47]  Sumit Roy,et al.  Two-dimensional equalization: theory and applications to high density magnetic recording , 1994, IEEE Trans. Commun..

[48]  Radford M. Neal,et al.  Near Shannon limit performance of low density parity check codes , 1996 .

[49]  Payam Pakzad,et al.  Kikuchi approximation method for joint decoding of LDPC codes and partial-response channels , 2006, IEEE Transactions on Communications.

[50]  A. Glavieux,et al.  Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1 , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[51]  Anthony J. Weiss,et al.  Generalized belief propagation receiver for near-optimal detection of two-dimensional channels with memory , 2004, Information Theory Workshop.

[52]  B. Leroux Stochastic Processes and Their Applications Maximum-likelihood Estimation for Hidden Markov Models Markov Chain * Consistency * Subadditive Ergodic Theorem * Identifiability * Entropy * Kullback-leibler Divergence * Shannon-mcmillan-breiman Theorem , 2022 .

[53]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[54]  Ido Kanter,et al.  Statistical mechanical aspects of joint source-channel coding , 2003 .

[55]  Joseph A. O'Sullivan,et al.  Iterative detection and decoding for separable two-dimensional intersymbol interference , 2003 .

[56]  D. Ornstein,et al.  The Shannon-McMillan-Breiman theorem for a class of amenable groups , 1983 .