Model Reductions for Inference: Generality of Pairwise, Binary, and Planar Factor Graphs

We offer a solution to the problem of efficiently translating algorithms between different types of discrete statistical model. We investigate the expressive power of three classes of model—those with binary variables, with pairwise factors, and with planar topology—as well as their four intersections. We formalize a notion of “simple reduction” for the problem of inferring marginal probabilities and consider whether it is possible to “simply reduce” marginal inference from general discrete factor graphs to factor graphs in each of these seven subclasses. We characterize the reducibility of each class, showing in particular that the class of binary pairwise factor graphs is able to simply reduce only positive models. We also exhibit a continuous “spectral reduction” based on polynomial interpolation, which overcomes this limitation. Experiments assess the performance of standard approximate inference algorithms on the outputs of our reductions.

[1]  Donald E. Knuth.  Introduction to combinatorial algorithms and boolean functions , 2008 .

[2]  K. Wagner.  Über eine Eigenschaft der ebenen Komplexe , 1937 .

[3]  Leslie Ann Goldberg,et al.  Inapproximability of the Tutte polynomial , 2006, STOC '07.

[4]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[5]  Leslie G. Valiant,et al.  The Complexity of Enumeration and Reliability Problems , 1979, SIAM J. Comput..

[6]  Kristian G. Olesen,et al.  An algebra of bayesian belief universes for knowledge-based systems , 1990, Networks.

[7]  P. W. Kasteleyn.  The Statistics of Dimers on a Lattice , 1961 .

[8]  Hilbert J. Kappen,et al.  Validity Estimates for Loopy Belief Propagation on Binary Real-world Networks , 2004, NIPS.

[9]  W. T. Tutte.  A THEOREM ON PLANAR GRAPHS , 1956 .

[10]  Endre Boros,et al.  Pseudo-Boolean optimization , 2002, Discret. Appl. Math..

[11]  Dror Weitz,et al.  Counting independent sets up to the tree threshold , 2006, STOC '06.

[12]  Enrique F. Castillo,et al.  Expert Systems and Probabilistic Network Models , 1996, Monographs in Computer Science.

[13]  Hilbert J. Kappen,et al.  Loop Corrected Belief Propagation , 2007, AISTATS.

[14]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[15]  M. Fisher.  On the Dimer Solution of Planar Ising Models , 1966 .

[16]  Joris M. Mooij,et al.  libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models , 2010, J. Mach. Learn. Res..

[17]  Leslie G. Valiant,et al.  The Complexity of Computing the Permanent , 1979, Theor. Comput. Sci..

[18]  Yee Whye Teh,et al.  Linear Response Algorithms for Approximate Inference in Graphical Models , 2004, Neural Computation.

[19]  Yee Whye Teh,et al.  Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation , 2001, UAI.

[20]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[21]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[22]  Judea Pearl,et al.  Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach , 1982, AAAI.

[23]  David Lichtenstein,et al.  Planar Formulae and Their Uses , 1982, SIAM J. Comput..

[24]  Sanjeev Arora,et al.  Computational Complexity: A Modern Approach , 2009 .

[25]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[26]  P. W. Kasteleyn.  The statistics of dimers on a lattice: I. The number of dimer arrangements on a quadratic lattice , 1961 .

[27]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.

[28]  Hilbert J. Kappen,et al.  On the properties of the Bethe approximation and loopy belief propagation on binary networks , 2004 .

[29]  Hiroshi Ishikawa,et al.  Higher-order clique reduction in binary graph cut , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[31]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[32]  Thomas J. Schaefer,et al.  The complexity of satisfiability problems , 1978, STOC.

[33]  Alexander T. Ihler,et al.  Accuracy Bounds for Belief Propagation , 2007, UAI.

[34]  Donald E. Knuth,et al.  The Art of Computer Programming, Volume 4, Fascicle 0: Introduction to Combinatorial Algorithms and Boolean Functions (Art of Computer Programming) , 2008 .

[35]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[36]  Martin E. Dyer,et al.  The Complexity of Weighted Boolean #CSP , 2009, SIAM J. Comput..

[37]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[38]  F. Barahona.  On the computational complexity of Ising spin glass models , 1982 .

[39]  Prasad Tetali,et al.  The correlation decay (CD) tree and strong spatial mixing in multi-spin systems , 2007 .

[40]  Michael Chertkov,et al.  Loop series for discrete statistical models on graphs , 2006, ArXiv.

[41]  Hilbert J. Kappen,et al.  Approximate Inference and Constrained Optimization , 2002, UAI.

[42]  Tommi S. Jaakkola,et al.  Approximate inference using planar graph decomposition , 2006, NIPS.

[43]  José Manuel Gutiérrez,et al.  Expert Systems and Probabiistic Network Models , 1996 .

[44]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[45]  Devavrat Shah,et al.  Message Passing for Maximum Weight Independent Set , 2008, IEEE Transactions on Information Theory.

[46]  Salil P. Vadhan,et al.  Computational Complexity , 2005, Encyclopedia of Cryptography and Security.

[47]  Umesh V. Vazirani,et al.  Quantum complexity theory , 1993, STOC.

[48]  Erik B. Sudderth,et al.  Loop Series and Bethe Variational Bounds in Attractive Graphical Models , 2007, NIPS.

[49]  Hector J. Levesque,et al.  A New Method for Solving Hard Satisfiability Problems , 1992, AAAI.

[50]  Michael Chertkov,et al.  Belief propagation and loop series on planar graphs , 2008, ArXiv.

[51]  D. Shah,et al.  Inference in Binary Pair-wise Markov Random Fields through Self-Avoiding Walks , 2006 .

[52]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[53]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Stephen A. Cook,et al.  The complexity of theorem-proving procedures , 1971, STOC.

[55]  C. Kuratowski.  Sur le problème des courbes gauches en Topologie , 1930 .

[56]  Kenji Fukumizu,et al.  New Graph Polynomials from the Bethe Approximation of the Ising Partition Function , 2009, Combinatorics, Probability and Computing.

[57]  Frederik Eaton.  A conditional game for comparing approximations , 2011, AISTATS.

[58]  Andrei A. Bulatov,et al.  The complexity of partition functions , 2005, Theor. Comput. Sci..

[59]  A. Montanari,et al.  How to compute loop corrections to the Bethe approximation , 2005, cond-mat/0506769.

[60]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[61]  P. W. Kasteleyn.  Dimer Statistics and Phase Transitions , 1963 .