Convexity Arguments for Efficient Minimization of the Bethe and Kikuchi Free Energies

Loopy and generalized belief propagation are popular algorithms for approximate inference in Markov random fields and Bayesian networks. Fixed points of these algorithms have been shown to correspond to extrema of the Bethe and Kikuchi free energy, both of which are approximations of the exact Helmholtz free energy. However, belief propagation does not always converge, which motivates approaches that explicitly minimize the Kikuchi/Bethe free energy, such as CCCP and UPS. Here we describe a class of algorithms that solves this typically non-convex constrained minimization problem through a sequence of convex constrained minimizations of upper bounds on the Kikuchi free energy. Intuitively one would expect tighter bounds to lead to faster algorithms, which is indeed convincingly demonstrated in our simulations. Several ideas are applied to obtain tight convex bounds that yield dramatic speed-ups over CCCP.

[1]  R. Jirousek,et al.  On the effective implementation of the iterative proportional fitting procedure , 1995 .

[2]  Tom Heskes,et al.  Stable Fixed Points of Loopy Belief Propagation Are Local Minima of the Bethe Free Energy , 2002, NIPS.

[3]  Michael I. Jordan,et al.  Variational Probabilistic Inference and the QMR-DT Network , 2011, J. Artif. Intell. Res..

[4]  John W. Fisher,et al.  Loopy Belief Propagation: Convergence and Effects of Message Errors , 2005, J. Mach. Learn. Res..

[5]  P. Hall.  On Representatives of Subsets , 1935 .

[6]  J. Darroch,et al.  Generalized Iterative Scaling for Log-Linear Models , 1972 .

[7]  Hilbert J. Kappen,et al.  Approximate Inference and Constrained Optimization , 2002, UAI.

[8]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[9]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[10]  Tom Heskes,et al.  Approximate Expectation Maximization , 2003, NIPS.

[11]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[12]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[13]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[14]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[15]  S. Katsura,et al.  On the Theory of Cooperative Phenomena , 1954 .

[16]  Kevin Murphy,et al.  Bayes net toolbox for Matlab , 1999 .

[17]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[18]  David Heckerman,et al.  A Tractable Inference Algorithm for Diagnosing Multiple Diseases , 2013, UAI.

[19]  Martin J. Wainwright,et al.  Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching , 2003, AISTATS.

[20]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[21]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[22]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[23]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[24]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[25]  Payam Pakzad,et al.  Estimation and Marginalization Using the Kikuchi Approximation Methods , 2005, Neural Computation.

[26]  Jeffrey C. Lagarias,et al.  Minimax and Hamiltonian Dynamics of Excitatory-Inhibitory Networks , 1997, NIPS.

[27]  T. Heskes.  Stable Fixed Points of Loopy Belief Propagation Are Minima of the Bethe Free Energy , 2002 .

[28]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[29]  Payam Pakzad,et al.  Belief Propagation and Statistical Physics , 2002 .

[30]  Yee Whye Teh,et al.  The Unified Propagation and Scaling Algorithm , 2001, NIPS.

[31]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[32]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[33]  David G. Luenberger,et al.  Linear and Nonlinear Programming: Second Edition , 2003 .

[34]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[35]  Rina Dechter,et al.  Iterative Join-Graph Propagation , 2002, UAI.

[36]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[37]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .