Convergence analysis for Guassian belief propagation: Dynamic behaviour of marginal covariances

Despite of its wide success in many distributed statistical learning applications, the well-known Gaussian belief propagation (BP) algorithm still lacks sufficient understanding at the theoretical level. This paper studies the convergence of Gaussian BP by analyzing the dynamic behaviour of the marginal covariances. We show, under a mild technical assumption, that the information matrices (i.e., the inverses of marginal covariances) are guaranteed to converge exponentially to positive-definite matrices. The convergence rate is explicitly characterized. This result is a key step to the understanding of the dynamic behaviour of the BP iterations.

[1]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[2]  Alain Glavieux,et al.  Reflections on the Prize Paper : "Near optimum error-correcting coding and decoding: turbo codes" , 1998 .

[3]  Riccardo Zecchina,et al.  Survey propagation as local equilibrium equations , 2003, ArXiv.

[4]  Hilbert J. Kappen,et al.  Sufficient Conditions for Convergence of the Sum–Product Algorithm , 2005, IEEE Transactions on Information Theory.

[5]  Minyue Fu,et al.  A new distributed state estimation technique for power networks , 2013, 2013 American Control Conference.

[6]  Yik-Chung Wu,et al.  On Convergence Conditions of Gaussian Belief Propagation , 2015, IEEE Transactions on Signal Processing.

[7]  Sekhar C. Tatikonda,et al.  Convergence of the sum-product algorithm , 2003, Proceedings 2003 IEEE Information Theory Workshop (Cat. No.03EX674).

[8]  Kazuyuki Tanaka.  Statistical-mechanical approach to image processing , 2002 .

[9]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message-Passing for Convex Optimization , 2010, IEEE Transactions on Information Theory.

[10]  Yik-Chung Wu,et al.  Convergence Analysis of the Variance in Gaussian Belief Propagation , 2014, IEEE Transactions on Signal Processing.

[11]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[12]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Dmitry M. Malioutov,et al.  Walk-Sums and Belief Propagation in Gaussian Graphical Models , 2006, J. Mach. Learn. Res..

[14]  John W. Fisher,et al.  Message Errors in Belief Propagation , 2004, NIPS.

[15]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[16]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[17]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[18]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[19]  John W. Fisher,et al.  Loopy Belief Propagation: Convergence and Effects of Message Errors , 2005, J. Mach. Learn. Res..

[20]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[21]  Andrea Montanari,et al.  Belief Propagation Based Multi--User Detection , 2005, ArXiv.

[22]  Minyue Fu,et al.  Stochastic analysis of turbo decoding , 2005, IEEE Transactions on Information Theory.

[23]  William T. Freeman,et al.  Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology , 1999, Neural Computation.

[24]  Defeng Huang,et al.  EM-Based Joint Channel Estimation and Detection for Frequency Selective Channels Using Gaussian Message Passing , 2011, IEEE Transactions on Signal Processing.

[25]  C MoallemiCiamac,et al.  Convergence of min-sum message-passing for convex optimization , 2010 .