Max-Product Belief Propagation for Linear Programming: Applications to Combinatorial Optimization

The max-product {belief propagation} (BP) is a popular message-passing heuristic for approximating a maximum-a-posteriori (MAP) assignment in a joint distribution represented by a graphical model (GM). In the past years, it has been shown that BP can solve a few classes of linear programming (LP) formulations to combinatorial optimization problems including maximum weight matching, shortest path and network flow, i.e., BP can be used as a message-passing solver for certain combinatorial optimizations. However, those LPs and corresponding BP analysis are very sensitive to underlying problem setups, and it has been not clear what extent these results can be generalized to. In this paper, we obtain a generic criteria that BP converges to the optimal solution of given LP, and show that it is satisfied in LP formulations associated to many classical combinatorial optimization problems including maximum weight perfect matching, shortest path, traveling salesman, cycle packing, vertex/edge cover and network flow.

[1]  Viktor K. Prasanna,et al.  Task Parallel Implementation of Belief Propagation in Factor Graphs , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum.

[2]  Dmitry M. Malioutov,et al.  Belief Propagation and LP Relaxation for Weighted Matching in General Graphs , 2011, IEEE Transactions on Information Theory.

[3]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[4]  Jinwoo Shin,et al.  A Graphical Transformation for Belief Propagation: Maximum Weight Matchings and Odd-Sized Cycles , 2013, NIPS.

[5]  Devavrat Shah,et al.  Maximum weight matching via max-product belief propagation , 2005, ISIT.

[6]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[7]  Yuval Rabani,et al.  Linear Programming , 2007, Handbook of Approximation Algorithms and Metaheuristics.

[8]  Joseph E. Gonzalez,et al.  Parallel Splash Belief Propagation , 2010 .

[9]  Meritxell Vinyals,et al.  Worst-case bounds on the quality of max-product fixed-points , 2010, NIPS.

[10]  David Gamarnik,et al.  Counting without sampling: new algorithms for enumeration problems using statistical physics , 2006, SODA '06.

[11]  Devavrat Shah,et al.  Message Passing for Maximum Weight Independent Set , 2008, IEEE Transactions on Information Theory.

[12]  Devavrat Shah,et al.  Belief Propagation: An Asymptotically Optimal Algorithm for the Random Assignment Problem , 2009, Math. Oper. Res..

[13]  Michael J. Todd,et al.  Polynomial Algorithms for Linear Programming , 1988 .

[14]  Alexander Schrijver,et al.  Combinatorial optimization. Polyhedra and efficiency. , 2003 .

[15]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[16]  S. Kak.  Information, physics, and computation , 1996 .

[17]  Edward H. Adelson,et al.  Belief Propagation and Revision in Networks with Loops , 1997 .

[18]  L. Khachiyan.  Polynomial algorithms in linear programming , 1980 .

[19]  Christian Borgs,et al.  Belief Propagation for Weighted b-Matchings on Arbitrary Graphs and its Relation to Linear Programs with Integer Solutions , 2007, SIAM J. Discret. Math..

[20]  N. Ruozzi,et al.  s-t paths using the min-sum algorithm , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[21]  Devavrat Shah,et al.  Counting Independent Sets Using the Bethe Approximation , 2011, SIAM J. Discret. Math..

[22]  N. Ma.  Task Parallel Implementation of Belief Propagation in Factor Graphs , 2012 .

[23]  Venkat Chandrasekaran,et al.  Complexity of Inference in Graphical Models , 2008, UAI.

[24]  Joseph M. Hellerstein,et al.  GraphLab: A New Framework For Parallel Machine Learning , 2010, UAI.

[25]  Bert Huang,et al.  Loopy Belief Propagation for Bipartite Maximum Weight b-Matching , 2007, AISTATS.

[26]  Péter Kovács,et al.  LEMON - an Open Source C++ Graph Template Library , 2011, WGT@ETAPS.

[27]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[28]  Devavrat Shah,et al.  Maximum weight matching via max-product belief propagation , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[29]  Thomas Zaslavsky,et al.  A simple algorithm that proves half-integrality of bidirected network programming , 2006 .

[30]  Nesa L'abbe Wu,et al.  Linear programming and extensions , 1981 .

[31]  John N. Tsitsiklis,et al.  Introduction to linear optimization , 1997, Athena scientific optimization and computation series.

[32]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.