Monte Carlo Algorithms for the Partition Function and Information Rates of Two-Dimensional Channels

The paper proposes Monte Carlo algorithms for the computation of the information rate of 2-D source/channel models. The focus of the paper is on binary-input channels with constraints on the allowed input configurations. The problem of numerically computing the information rate, and even the noiseless capacity, of such channels has so far remained largely unsolved. Both problems can be reduced to computing a Monte Carlo estimate of a partition function. The proposed algorithms use tree-based Gibbs sampling and multilayer (multitemperature) importance sampling. The viability of the proposed algorithms is demonstrated by simulation results.

[1]  Paul H. Siegel,et al.  Codes for Digital Recorders , 1998, IEEE Trans. Inf. Theory.

[2]  Mehdi Molkaraie,et al.  Ieee Transactions on Communications, Accepted for Publication Generalized Belief Propagation for the Noiseless Capacity and Information Rates of Run-length Limited Constraints , 2022 .

[3]  A. Kavcic,et al.  The Feasibility of Magnetic Recording at 10 Terabits Per Square Inch on Conventional Media , 2009, IEEE Transactions on Magnetics.

[4]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[5]  Claude E. Shannon,et al.  A Mathematical Theory of Communications , 1948 .

[6]  Gerasimos Potamianos,et al.  Stochastic approximation algorithms for partition function estimation of Gibbs random fields , 1997, IEEE Trans. Inf. Theory.

[7]  Schouhamer Immink,et al.  Codes for mass data storage systems , 2004 .

[8]  Ron M. Roth,et al.  Introduction to Coding Theory , 2019, Discrete Mathematics.

[9]  H. Katzgraber.  Introduction to Monte Carlo Methods , 2009, 0905.1629.

[10]  Kenneth Zeger,et al.  On the capacity of two-dimensional run-length constrained channels , 1999, IEEE Trans. Inf. Theory.

[11]  Charles H. Bennett,et al.  Efficient estimation of free energy differences from Monte Carlo data , 1976 .

[12]  S. E. Ahmed,et al.  Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 2008, Technometrics.

[13]  Paul H. Siegel,et al.  On the symmetric information rate of two-dimensional finite state ISI channels , 2003, Proceedings 2003 IEEE Information Theory Workshop (Cat. No.03EX674).

[14]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Tuvi Etzion,et al.  The Positive Capacity Region of Two-Dimensional Run-Length-Constrained Channels , 2006, IEEE Transactions on Information Theory.

[16]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[17]  Hans-Andrea Loeliger,et al.  On the information rate of binary-input channels with memory , 2001, ICC 2001. IEEE International Conference on Communications. Conference Record (Cat. No.01CH37240).

[18]  Radford M. Neal.  Annealed importance sampling , 1998, Stat. Comput..

[19]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[20]  H.-A. Loeliger,et al.  An introduction to factor graphs , 2004, IEEE Signal Process. Mag..

[21]  Richard E. Blahut,et al.  The Capacity and Coding Gain of Certain Checkerboard Codes , 1998, IEEE Trans. Inf. Theory.

[22]  Bane Vasic,et al.  Coding and Signal Processing for Magnetic Recording Systems , 2004 .

[23]  Hans-Andrea Loeliger,et al.  Simulation-based estimation of the partition function and the information rate of two-dimensional models , 2008, 2008 IEEE International Symposium on Information Theory.

[24]  Mehdi Molkaraie,et al.  Generalized belief propagation algorithm for the capacity of multi-dimensional run-length limited constraints , 2010, 2010 IEEE International Symposium on Information Theory.

[25]  Yuan Zhou.  Introduction to Coding Theory , 2010 .

[26]  Paul H. Siegel,et al.  On the achievable information rates of finite state ISI channels , 2001, GLOBECOM'01. IEEE Global Telecommunications Conference (Cat. No.01CH37270).

[27]  Ido Tal,et al.  Concave programming upper bounds on the capacity of 2-D constraints , 2009, 2009 IEEE International Symposium on Information Theory.

[28]  Paul H. Siegel,et al.  On the symmetric information rate of two-dimensional finite-state ISI channels , 2006, IEEE Transactions on Information Theory.

[29]  Hans-Andrea Loeliger,et al.  Estimating the partition function of 2-D fields and the capacity of constrained noiseless 2-D channels using tree-based Gibbs sampling , 2009, 2009 IEEE Information Theory Workshop.

[30]  Radford M. Neal.  Estimating Ratios of Normalizing Constants Using Linked Importance Sampling , 2005, math/0511216.

[31]  Herbert S. Wilf,et al.  The Number of Independent Sets in a Grid Graph , 1998, SIAM J. Discret. Math..

[32]  C. Jarzynski.  Nonequilibrium Equality for Free Energy Differences , 1996, cond-mat/9610209.

[33]  Hisashi Ito,et al.  Zero capacity region of multidimensional run length constraints , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[34]  Y. Ogata,et al.  Estimation of interaction potentials of spatial point patterns through the maximum likelihood procedure , 1981 .

[35]  Shlomo Shamai,et al.  Discrete-Input Two-Dimensional Gaussian Channels With Memory: Estimation and Information Rates Via Graphical Models and Statistical Mechanics , 2008, IEEE Transactions on Information Theory.

[36]  John Odentrantz,et al.  Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues , 2000, Technometrics.

[37]  Mark Jerrum,et al.  Polynomial-Time Approximation Algorithms for the Ising Model , 1990, SIAM J. Comput..

[38]  Nando de Freitas,et al.  From Fields to Trees , 2004, UAI.

[39]  Kenneth Zeger,et al.  Capacity bounds for the three-dimensional (0, 1) run length limited channel , 2000, IEEE Trans. Inf. Theory.

[40]  Xiao-Li Meng,et al.  SIMULATING RATIOS OF NORMALIZING CONSTANTS VIA A SIMPLE IDENTITY: A THEORETICAL EXPLORATION , 1996 .

[41]  Wei Zeng,et al.  Simulation-Based Computation of Information Rates for Channels With Memory , 2006, IEEE Transactions on Information Theory.