Iterative Decoding of Compound Codes by Probability Propagation in Graphical Models

We present a unified graphical model framework for describing compound codes and deriving iterative decoding algorithms. After reviewing a variety of graphical models (Markov random fields, Tanner graphs, and Bayesian networks), we derive a general distributed marginalization algorithm for functions described by factor graphs. From this general algorithm, Pearl's (1986) belief propagation algorithm is easily derived as a special case. We point out that iterative decoding algorithms for various codes, including "turbo decoding" of parallel-concatenated convolutional codes, may be viewed as probability propagation in a graphical model of the code. We focus on Bayesian network descriptions of codes, which give a natural input/state/output/channel description of a code and channel, and we indicate how iterative decoders can be developed for parallel-and serially concatenated coding systems, product codes, and low-density parity-check codes.

[1]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[2]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[3]  C. Preston.  Gibbs States on Countable Sets , 1974 .

[4]  John Cocke,et al.  Optimal decoding of linear codes for minimizing symbol error rate (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[5]  J. Laurie Snell,et al.  Markov Random Fields and Their Applications , 1980 .

[6]  Robert Michael Tanner,et al.  A recursive approach to low complexity codes , 1981, IEEE Trans. Inf. Theory.

[7]  V. Isham.  An Introduction to Spatial Point Processes and Markov Random Fields , 1981 .

[8]  Geoffrey E. Hinton,et al.  Learning and relearning in Boltzmann machines , 1986 .

[9]  Judea Pearl,et al.  Fusion, Propagation, and Structuring in Belief Networks , 1986, Artif. Intell..

[10]  Dimitri P. Bertsekas,et al.  Data Networks , 1986 .

[11]  S. Verdú,et al.  Abstract dynamic programming models under commutativity conditions , 1987 .

[12]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[13]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[14]  Richard E. Neapolitan,et al.  Probabilistic reasoning in expert systems - theory and algorithms , 2012 .

[15]  Dimitri P. Bertsekas,et al.  Data networks (2nd ed.) , 1992 .

[16]  A. Glavieux,et al.  Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1 , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[17]  Peter Adam Hoeher,et al.  Separable MAP "filters" for the decoding of product and concatenated codes , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[18]  G. David Forney.  Trellises Old and New , 1994 .

[19]  J. Massey,et al.  Communications and Cryptography: Two Sides of One Tapestry , 1994 .

[20]  Hans-Andrea Loeliger,et al.  Codes and iterative decoding on general graphs , 1995, Eur. Trans. Telecommun..

[21]  David J. C. MacKay,et al.  Good Codes Based on Very Sparse Matrices , 1995, IMACC.

[22]  Brendan J. Frey,et al.  Probability Propagation and Iterative Decoding , 1996 .

[23]  Radford M. Neal,et al.  Near Shannon Limit Performance of Low Density Parity Check Codes , 1996 .

[24]  Robert J. McEliece,et al.  On the BCJR trellis for linear block codes , 1996, IEEE Trans. Inf. Theory.

[25]  Radford M. Neal,et al.  Near Shannon limit performance of low density parity check codes , 1996 .

[26]  Sergio Benedetto,et al.  Serial concatenation of block and convolutional codes , 1996 .

[27]  Sergio Benedetto,et al.  Iterative decoding of serially concatenated convolutional codes , 1996 .

[28]  Joachim Hagenauer,et al.  Iterative decoding of binary block and convolutional codes , 1996, IEEE Trans. Inf. Theory.

[29]  Niclas Wiberg,et al.  Codes and Decoding on General Graphs , 1996 .

[30]  Robert J. McEliece,et al.  A general algorithm for distributing information in a graph , 1997, Proceedings of IEEE International Symposium on Information Theory.

[31]  Brendan J. Frey,et al.  Concurrent turbo-decoding , 1997, Proceedings of IEEE International Symposium on Information Theory.

[32]  Brendan J. Frey,et al.  Bayesian networks for pattern classification, data compression, and channel coding , 1997 .

[33]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[34]  Brendan J. Frey,et al.  Graphical Models for Machine Learning and Digital Communication , 1998 .

[35]  Michael I. Jordan.  Graphical Models , 2003 .

[36]  David J. C. MacKay,et al.  Good Error-Correcting Codes Based on Very Sparse Matrices , 1997, IEEE Trans. Inf. Theory.