An Edge Deletion Semantics for Belief Propagation and its Practical Impact on Approximation Quality

We show in this paper that the influential algorithm of iterative belief propagation can be understood in terms of exact inference on a polytree, which results from deleting enough edges from the original network. We show that deleting edges implies adding new parameters into a network, and that the iterations of belief propagation are searching for values of these new parameters which satisfy intuitive conditions that we characterize. The new semantics lead to the following question: Can one improve the quality of approximations computed by belief propagation by recovering some of the deleted edges, while keeping the network easy enough for exact inference? We show in this paper that the answer is yes, leading to another question: How do we choose which edges to recover? To answer, we propose a specific method based on mutual information which is motivated by the edge deletion semantics. Empirically, we provide experimental results showing that the quality of approximations can be improved without incurring much additional computational cost. We also show that recovering certain edges with low mutual information may not be worthwhile as they increase the computational complexity, without necessarily improving the quality of approximations.

[1]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[2]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[3]  Adnan Darwiche,et al.  On Bayesian Network Approximation by Edge Deletion , 2005, UAI.

[4]  Rina Dechter,et al.  Iterative Join-Graph Propagation , 2002, UAI.

[5]  Adnan Darwiche,et al.  On the Revision of Probabilistic Beliefs using Uncertain Evidence , 2003, IJCAI.

[6]  Adnan Darwiche,et al.  Compiling Bayesian Networks with Local Structure , 2005, IJCAI.

[7]  Adnan Darwiche,et al.  Recursive conditioning , 2001, Artif. Intell..

[8]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[9]  Rina Dechter,et al.  Bucket elimination: A unifying framework for probabilistic inference , 1996, UAI.

[10]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[11]  Brendan J. Frey,et al.  A Revolution: Belief Propagation in Graphs with Cycles , 1997, NIPS.

[12]  Yee Whye Teh,et al.  Structured Region Graphs: Morphing EP into GBP , 2005, UAI.

[13]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[14]  Adnan Darwiche,et al.  A Variational Approach for Approximating Bayesian Networks by Edge Deletion , 2006, UAI.

[15]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[16]  Martin J. Wainwright,et al.  Tree-based reparameterization framework for analysis of sum-product and related algorithms , 2003, IEEE Trans. Inf. Theory.

[17]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[18]  Nevin Lianwen Zhang,et al.  Exploiting Causal Independence in Bayesian Network Inference , 1996, J. Artif. Intell. Res..

[19]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.