Belief Propagation , Mean-field , and Bethe approximations

This chapter describes methods for estimating the marginals and maximum a posteriori (MAP) estimates of probability distributions defined over graphs by approximate methods including Mean Field Theory (MFT), variational methods, and belief propagation. These methods typically formulate this problem in terms of minimizing a free energy function of pseudomarginals. They differ by the design of the free energy and the choice of algorithm to minimize it. These algorithms can often be interpreted in terms of message passing. In many cases, the free energy has a dual formulation and the algorithms are defined over the dual variables (e.g., the messages in belief propagation). The quality of performance depends on the types of free energies used – specifically how well they approximate the log partition function of the probability distribution – and whether there are suitable algorithms for finding their minima. We start in section (II) by introducing two types of Markov Field models that are often used in computer vision. We proceed to define MFT/variational methods in section (III), whose free energies are lower bounds of the log partition function, and describe how inference can be done by expectation-maximization, steepest descent, or discrete iterative algorithms. The following section (IV) describes message passing algorithms, such as belief propagation and its generalizations, which can be related to free energy functions (and dual variables). Finally in section (V) we describe how these methods relate to Markov Chain Monte Carlo (MCMC) approaches, which gives a different way to think of these methods and which can lead to novel algorithms.