Belief Propagation Neural Networks

Learned neural solvers have successfully been used to solve combinatorial optimization and decision problems. More general counting variants of these problems, however, are still largely solved with hand-crafted solvers. To bridge this gap, we introduce belief propagation neural networks (BPNNs), a class of parameterized operators that operate on factor graphs and generalize Belief Propagation (BP). In its strictest form, a BPNN layer (BPNN-D) is a learned iterative operator that provably maintains many of the desirable properties of BP for any choice of the parameters. Empirically, we show that by training BPNN-D learns to perform the task better than the original BP: it converges 1.7x faster on Ising models while providing tighter bounds. On challenging model counting problems, BPNNs compute estimates 100's of times faster than state-of-the-art handcrafted methods, while returning an estimate of comparable quality.

[1]  Zayd Hammoudeh,et al.  Fast and Flexible Probabilistic Model Counting , 2018, SAT.

[2]  Stefano Ermon,et al.  Learning Neural PDE Solvers with Convergence Guarantees , 2019, ICLR.

[3]  David L. Dill,et al.  Learning a SAT Solver from Single-Bit Supervision , 2018, ICLR.

[4]  森 立平.  New understanding of the Bethe approximation and the replica method , 2013 .

[5]  Christian J. Muise,et al.  Dsharp: Fast d-DNNF Compilation with sharpSAT , 2012, Canadian Conference on AI.

[6]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[7]  Ryuhei Mori,et al.  New Understanding of the Bethe Approximation and the Replica Method , 2013, ArXiv.

[8]  Dan Roth,et al.  On the Hardness of Approximate Reasoning , 1993, IJCAI.

[9]  Lisa Zhang,et al.  Inference in Probabilistic Graphical Models by Graph Neural Networks , 2018, 2019 53rd Asilomar Conference on Signals, Systems, and Computers.

[10]  Axel Legay,et al.  Scalable Approximation of Quantitative Information Flow in Programs , 2018, VMCAI.

[11]  Moshe Y. Vardi,et al.  Counting-Based Reliability Estimation for Power-Transmission Grids , 2017, AAAI.

[12]  Bart Selman,et al.  Model Counting: A New Strategy for Obtaining Good Bounds , 2006, AAAI.

[13]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[14]  Max Welling,et al.  Neural Enhanced Belief Propagation on Factor Graphs , 2020, AISTATS.

[15]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[16]  Claude Castelluccia,et al.  Extending SAT Solvers to Cryptographic Problems , 2009, SAT.

[17]  Mate Soos,et al.  BIRD: Engineering an Efficient CNF-XOR SAT Solver and Its Applications to Approximate Model Counting , 2019, AAAI.

[18]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[19]  Guy Van den Broeck,et al.  Hashing-Based Approximate Probabilistic Inference in Hybrid Domains , 2015, UAI.

[20]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[21]  Larry J. Stockmeyer,et al.  The complexity of approximate counting , 1983, STOC.

[22]  H. Bethe.  Statistical Theory of Superlattices , 1935 .

[23]  Leslie G. Valiant,et al.  The Complexity of Enumeration and Reliability Problems , 1979, SIAM J. Comput..

[24]  Sam Wiseman,et al.  Amortized Bethe Free Energy Minimization for Learning MRFs , 2019, NeurIPS.

[25]  Joan Bruna,et al.  Community Detection with Graph Neural Networks , 2017, 1705.08415.

[26]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[27]  Richard M. Karp,et al.  Monte-Carlo Approximation Algorithms for Enumeration Problems , 1989, J. Algorithms.

[28]  Thomas Lukasiewicz,et al.  Learning to Reason: Leveraging Neural Networks for Approximate DNF Counting , 2019, AAAI.

[29]  Supratik Chakraborty,et al.  Algorithmic Improvements in Approximate Counting for Probabilistic Inference: From Linear to Logarithmic SAT Calls , 2016, IJCAI.

[30]  Cristopher Moore,et al.  Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  Leslie G. Valiant,et al.  Random Generation of Combinatorial Structures from a Uniform Distribution , 1986, Theor. Comput. Sci..

[32]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[33]  Mihir Bellare,et al.  Uniform Generation of NP-Witnesses Using an NP-Oracle , 2000, Inf. Comput..

[34]  Tommi S. Jaakkola,et al.  On the Partition Function and Random Maximum A-Posteriori Perturbations , 2012, ICML.

[35]  Sharad Malik,et al.  On computing minimal independent support and its applications to sampling and counting , 2015, Constraints.

[36]  David Duvenaud,et al.  Invertible Residual Networks , 2018, ICML.

[37]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[38]  Joris M. Mooij,et al.  libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models , 2010, J. Mach. Learn. Res..

[39]  Michael I. Jordan,et al.  Variational Probabilistic Inference and the QMR-DT Network , 2011, J. Artif. Intell. Res..

[40]  Nicholas Ruozzi,et al.  The Bethe Partition Function of Log-supermodular Graphical Models , 2012, NIPS.

[41]  Dimitris Achlioptas,et al.  Probabilistic Model Counting with Short XORs , 2017, SAT.

[42]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[43]  Daniel Tarlow,et al.  Learning to Pass Expectation Propagation Messages , 2013, NIPS.

[44]  Dimitris Achlioptas,et al.  Stochastic Integration via Error-Correcting Codes , 2015, UAI.

[45]  Bart Selman,et al.  Taming the Curse of Dimensionality: Discrete Integration by Hashing and Optimization , 2013, ICML.

[46]  Bart Selman,et al.  Low-density Parity Constraints for Hashing-Based Discrete Integration , 2014, ICML.

[47]  D. Chandler,et al.  Introduction To Modern Statistical Mechanics , 1987 .

[48]  Luís C. Lamb,et al.  Learning to Solve NP-Complete Problems - A Graph Neural Network for the Decision TSP , 2018, AAAI.

[49]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[50]  R. Baxter.  Exactly solved models in statistical mechanics , 1982 .

[51]  Frederic Koehler,et al.  Fast Convergence of Belief Propagation to Global Optima: Beyond Correlation Decay , 2019, NeurIPS.