Message-Passing Algorithms: Reparameterizations and Splittings

The max-product algorithm, a local message-passing scheme that attempts to compute the most probable assignment (MAP) of a given probability distribution, has been successfully employed as a method of approximate inference for applications arising in coding theory, computer vision, and machine learning. However, the max-product algorithm is not guaranteed to converge, and if it does, it is not guaranteed to recover the MAP assignment. Alternative convergent message-passing schemes have been proposed to overcome these difficulties. This paper provides a systematic study of such message-passing algorithms that extends the known results by exhibiting new sufficient conditions for convergence to local and/or global optima, providing a combinatorial characterization of these optima based on graph covers, and describing a new convergent and correct message-passing algorithm whose derivation unifies many of the known convergent message-passing algorithms. While convergent and correct message-passing algorithms represent a step forward in the analysis of max-product style message-passing algorithms, the conditions needed to guarantee convergence to a global optimum can be too restrictive in both theory and practice. This limitation of convergent and correct message-passing schemes is characterized by graph covers and illustrated by example.

[1]  Benjamin Van Roy,et al.  Convergence of the Min-Sum Algorithm for Convex Optimization , 2007, 0705.4253.

[2]  Tom Heskes,et al.  Fractional Belief Propagation , 2002, NIPS.

[3]  Tamir Hazan,et al.  Norm-Product Belief Propagation: Primal-Dual Message-Passing for Approximate Inference , 2009, IEEE Transactions on Information Theory.

[4]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message-Passing for Convex Optimization , 2010, IEEE Transactions on Information Theory.

[5]  N. Ruozzi,et al.  Graph covers and quadratic minimization , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[6]  Alexander Schrijver,et al.  Theory of linear and integer programming , 1986, Wiley-Interscience series in discrete mathematics and optimization.

[7]  Benjamin Van Roy,et al.  Convergence of the Min-Sum Message Passing Algorithm for Quadratic Optimization , 2006, ArXiv.

[8]  KolmogorovVladimir.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006 .

[9]  Martin J. Wainwright,et al.  Using linear programming to Decode Binary linear codes , 2005, IEEE Transactions on Information Theory.

[10]  Devavrat Shah,et al.  Message Passing for Maximum Weight Independent Set , 2008, IEEE Transactions on Information Theory.

[11]  David Burshtein,et al.  Iterative Approximate Linear Programming Decoding of LDPC Codes With Linear Complexity , 2008, IEEE Transactions on Information Theory.

[12]  Devavrat Shah,et al.  Tightness of LP via Max-product Belief Propagation , 2005, cs/0508097.

[13]  Tomás Werner,et al.  A Linear Programming Approach to Max-Sum Problem: A Review , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[15]  Stark C. Draper,et al.  Divide and Concur and Difference-Map BP Decoders for LDPC Codes , 2011, IEEE Transactions on Information Theory.

[16]  Marc Pollefeys,et al.  Distributed message passing for large scale graphical models , 2011, CVPR 2011.

[17]  Devavrat Shah,et al.  Maximum weight matching via max-product belief propagation , 2005, ISIT.

[18]  Martin J. Wainwright,et al.  Message-passing for Graph-structured Linear Programs: Proximal Methods and Rounding Schemes , 2010, J. Mach. Learn. Res..

[19]  Sekhar Tatikonda,et al.  Unconstrained minimization of quadratic functions via min-sum , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[20]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[21]  Amir Globerson,et al.  Convergent message passing algorithms - a unifying view , 2009, UAI.

[22]  Tommi S. Jaakkola,et al.  Fixing Max-Product: Convergent Message Passing Algorithms for MAP LP-Relaxations , 2007, NIPS.

[23]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[24]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  P. Vontobel,et al.  On the Relationship between Linear Programming Decoding and Min-Sum Algorithm Decoding , 2004 .

[26]  Judea Pearl,et al.  Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach , 1982, AAAI.

[27]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[28]  Yair Weiss,et al.  MAP Estimation, Linear Programming and Belief Propagation with Convex Free Energies , 2007, UAI.

[29]  Nathan Axvig.  LP Pseudocodewords of Cycle Codes are Half-Integral , 2012, ArXiv.

[30]  P. Vontobel,et al.  Graph-Cover Decoding and Finite-Length Analysis of Message-Passing Iterative Decoding of LDPC Codes , 2005, ArXiv.

[31]  Naum Zuselevich Shor,et al.  Minimization Methods for Non-Differentiable Functions , 1985, Springer Series in Computational Mathematics.

[32]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message Passing for Quadratic Optimization , 2006, IEEE Transactions on Information Theory.

[33]  Pascal O. Vontobel,et al.  Counting in Graph Covers: A Combinatorial Characterization of the Bethe Entropy Function , 2010, IEEE Transactions on Information Theory.

[34]  Martin J. Wainwright,et al.  Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching , 2003, AISTATS.

[35]  A. Glavieux,et al.  Near Shannon limit error-correcting coding and decoding: Turbo-codes. 1 , 1993, Proceedings of ICC '93 - IEEE International Conference on Communications.

[36]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[37]  Martin J. Wainwright,et al.  On the Optimality of Tree-reweighted Max-product Message-passing , 2005, UAI.

[38]  Tommi S. Jaakkola,et al.  Tree Block Coordinate Descent for MAP in Graphical Models , 2009, AISTATS.

[39]  N. Ruozzi,et al.  s-t paths using the min-sum algorithm , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[40]  Martin J. Wainwright,et al.  MAP estimation via agreement on (hyper)trees: Message-passing and linear programming , 2005, ArXiv.

[41]  Martin J. Wainwright,et al.  Tree consistency and bounds on the performance of the max-product algorithm and its generalizations , 2004, Stat. Comput..

[42]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[43]  Ralf Koetter,et al.  On low-complexity linear-programming decoding of LDPC codes , 2007, Eur. Trans. Telecommun..

[44]  Dana Angluin,et al.  Finite common coverings of pairs of regular graphs , 1981, J. Comb. Theory B.

[45]  Nikos Komodakis,et al.  MRF Optimization via Dual Decomposition: Message-Passing Revisited , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[46]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[47]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[48]  Ralf Koetter,et al.  Towards Low-Complexity Linear-Programming Decoding , 2006, ArXiv.

[49]  Pascal O. Vontobel,et al.  A combinatorial characterization of the Bethe and the Kikuchi partition functions , 2011, 2011 Information Theory and Applications Workshop.

[50]  J. WainwrightM.,et al.  MAP estimation via agreement on trees , 2005 .

[51]  Dmitry M. Malioutov,et al.  Walk-Sums and Belief Propagation in Gaussian Graphical Models , 2006, J. Mach. Learn. Res..

[52]  Deanna Dreher.  Cycles in Graphs and Covers , 2012, SIAM J. Discret. Math..