Accuracy of Loopy belief propagation in Gaussian models

This paper considers the loopy belief propagation (LBP) algorithm applied to Gaussian graphical models. It is known for Gaussian belief propagation that, if LBP converges, LBP computes the exact posterior means but incorrect variances. In this paper, we analytically derive the posterior variances for some special structured graphs and clarify the accuracy of LBP. For the graphs of a single cycle, we derive a rigorous solution for the posterior variances and thereby find the quantity that determines the accuracy of LBP. Based on this result, we state a necessary condition for LBP convergence. The quantity above also plays an important role in graphs of a single cycle with arbitrary trees. For arbitrary topological graphs, we consider the situation where correlations between any pair of nodes are comparatively small and show analytically the principal values that determine the accuracy of LBP.

[1]  Shun-ichi Amari,et al.  Information geometry of turbo and low-density parity-check codes , 2004, IEEE Transactions on Information Theory.

[2]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[3]  Martin J. Wainwright,et al.  Embedded trees: estimation of Gaussian Processes on graphs with cycles , 2004, IEEE Transactions on Signal Processing.

[4]  Sumio Watanabe,et al.  Singularities in mixture models and upper bounds of stochastic complexity , 2003, Neural Networks.

[5]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[6]  Sumio Watanabe.  Algebraic geometrical methods for hierarchical learning machines , 2001, Neural Networks.

[7]  Kazuho Watanabe,et al.  Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation , 2006, J. Mach. Learn. Res..

[8]  Kazuyuki Tanaka.  Statistical-mechanical approach to image processing , 2002 .

[9]  Shun-ichi Amari,et al.  Information geometry for turbo decoding , 2005, Systems and Computers in Japan.

[10]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[11]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[12]  D.J.C. MacKay,et al.  Good error-correcting codes based on very sparse matrices , 1997, Proceedings of IEEE International Symposium on Information Theory.

[13]  William T. Freeman,et al.  Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology , 1999, Neural Computation.

[14]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .