Constructing free-energy approximations and generalized belief propagation algorithms

Important inference problems in statistical physics, computer vision, error-correcting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixed points correspond to the stationary points of the Bethe approximation of the free energy for a factor graph. We explain how to obtain region-based free energy approximations that improve the Bethe approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a "valid" or "maxent-normal" approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the "Bethe method", the "junction graph method", the "cluster variation method", and the "region graph method". Finally, we explain how to tell whether a region-based approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.

[1]  H. Bethe.  Statistical Theory of Superlattices , 1935 .

[2]  E. M.,et al.  Statistical Mechanics , 2021, Manual for Theoretical Chemistry.

[3]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[4]  J. Hijmans,et al.  An approximation method for order-disorder problems. V , 1954 .

[5]  J. Hijmans,et al.  An approximation method for order-disorder problems III , 1954 .

[6]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[7]  Andrew J. Viterbi,et al.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.

[8]  A.H. Haddad,et al.  Applied optimal estimation , 1976, Proceedings of the IEEE.

[9]  S. Kirkpatrick,et al.  Infinite-ranged models of spin-glasses , 1978 .

[10]  S. Fujiki,et al.  Frustration effect on the d-dimensional Ising spin glass. II. Existence of the spin glass phase , 1980 .

[11]  R. Baxter.  Exactly solved models in statistical mechanics , 1982 .

[12]  Gerard Toulouse,et al.  Theory of the frustration effect in spin glasses: I , 1986 .

[13]  R. Stanley.  What Is Enumerative Combinatorics , 1986 .

[14]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[15]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[16]  Shirley Dex,et al.  JR 旅客販売総合システム（マルス）における運用及び管理について , 1991 .

[17]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[18]  T. Morita.  Cluster Variation Method for Non-Uniform Ising and Heisenberg Models and Spin-Pair Correlation Function , 1991 .

[19]  A. Glavieux,et al.  Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1 , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[20]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[21]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[22]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[23]  Brendan J. Frey,et al.  Graphical Models for Machine Learning and Digital Communication , 1998 .

[24]  Michael I. Jordan.  Graphical Models , 2003 .

[25]  Robert Cowell,et al.  Advanced Inference in Bayesian Networks , 1999, Learning in Graphical Models.

[26]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[27]  Robert J. McEliece,et al.  The generalized distributive law , 2000, IEEE Trans. Inf. Theory.

[28]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[29]  G. Forney,et al.  Codes on graphs: normal realizations , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[30]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[31]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[32]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[33]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[34]  D. Mackay.  A conversation about the Bethe free energy and sum-product , 2001 .

[35]  W. Freeman,et al.  Bethe free energy, Kikuchi approximations, and belief propagation algorithms , 2001 .

[36]  A. Yuille.  A Double-Loop Algorithm to Minimize the Bethe and Kikuchi Free Energies , 2001 .

[37]  Yee Whye Teh,et al.  Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation , 2001, UAI.

[38]  Hilbert J. Kappen,et al.  Novel iteration schemes for the Cluster Variation Method , 2001, NIPS.

[39]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[40]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[41]  Emina Soljanin,et al.  AN ALGEBRAIC DESCRIPTION OF ITERATIVE DECODING SCHEMES , 2001 .

[42]  Tom Heskes,et al.  Stable Fixed Points of Loopy Belief Propagation Are Local Minima of the Bethe Free Energy , 2002, NIPS.

[43]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[44]  Payam Pakzad,et al.  Belief Propagation and Statistical Physics , 2002 .

[45]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[46]  M. Mézard,et al.  Random K-satisfiability problem: from an analytic solution to an efficient algorithm. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[47]  Tom Heskes,et al.  Fractional Belief Propagation , 2002, NIPS.

[48]  Frank Wolter,et al.  Exploring Artificial Intelligence in the New Millenium , 2002 .

[49]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[50]  Yuan Qi,et al.  Tree-structured Approximations by Expectation Propagation , 2003, NIPS.

[51]  Jonathan Harel,et al.  Poset belief propagation-experimental results , 2003, IEEE International Symposium on Information Theory, 2003. Proceedings..

[52]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[53]  Justin Dauwels,et al.  On Structured-Summary Propagation, LFSR Synchronization, and Low-Complexity Trellis Decoding , 2003 .

[54]  Martin J. Wainwright,et al.  Tree-based reparameterization framework for analysis of sum-product and related algorithms , 2003, IEEE Trans. Inf. Theory.

[55]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[56]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[57]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[58]  M. Mézard,et al.  Survey propagation: An algorithm for satisfiability , 2005 .

[59]  Payam Pakzad,et al.  Estimation and Marginalization Using the Kikuchi Approximation Methods , 2005, Neural Computation.

[60]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[61]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.