A visual introduction to Gaussian Belief Propagation

In this article, we present a visual introduction to Gaussian Belief Propagation (GBP), an approximate probabilistic inference algorithm that operates by passing messages between the nodes of arbitrarily structured factor graphs. A special case of loopy belief propagation, GBP updates rely only on local information and will converge independently of the message schedule. Our key argument is that, given recent trends in computing hardware, GBP has the right computational properties to act as a scalable distributed probabilistic inference framework for future machine learning systems. Figure 1: INTERACTIVE FIGURE – view at https://gaussianbp.github.io. Gaussian Belief Propagation performs probabilistic inference iteratively and is convergent even when messages are passed randomly through the graph. Here GBP is applied to a geometric grid alignment problem. See an interactive version of this figure later in the article. 1 ar X iv :2 10 7. 02 30 8v 1 [ cs .A I] 5 J ul 2 02 1

[1]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[2]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[3]  Odest Chadwicke Jenkins,et al.  Differentiable Nonparametric Belief Propagation , 2021, ArXiv.

[4]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[5]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[6]  Danny Bickson,et al.  Gaussian Belief Propagation: Theory and Aplication , 2008, 0811.2518.

[7]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[8]  Frank Dellaert,et al.  Factor Graphs: Exploiting Structure in Robotics , 2021, Annu. Rev. Control. Robotics Auton. Syst..

[9]  Timothy D Barfoot,et al.  Fundamental Linear Algebra Problem of Gaussian Inference , 2020, ArXiv.

[10]  Andrew McCallum,et al.  Improved Dynamic Schedules for Belief Propagation , 2007, UAI.

[11]  Frank Dellaert,et al.  Factor Graphs for Robot Perception , 2017, Found. Trends Robotics.

[12]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[13]  Andrew J. Davison,et al.  FutureMapping: The Computational Structure of Spatial AI Systems , 2018, ArXiv.

[14]  Bhaskara Marthi,et al.  A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs , 2017, Science.

[15]  Frank Dellaert,et al.  Loopy SAM , 2007, IJCAI.

[16]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[17]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[18]  Vivienne Sze,et al.  Efficient Processing of Deep Neural Networks: A Tutorial and Survey , 2017, Proceedings of the IEEE.

[19]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[20]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[21]  N. Whitman.  A bitter lesson. , 1999, Academic medicine : journal of the Association of American Medical Colleges.

[22]  Michael I. Jordan.  Graphical Models , 2003 .

[23]  Andrew J. Davison,et al.  DeepFactors: Real-Time Probabilistic Dense Monocular SLAM , 2020, IEEE Robotics and Automation Letters.

[24]  Andrew J. Davison,et al.  FutureMapping 2: Gaussian Belief Propagation for Spatial AI , 2019, ArXiv.

[25]  Wolfram Burgard,et al.  G2o: A general framework for graph optimization , 2011, 2011 IEEE International Conference on Robotics and Automation.

[26]  Georgios Evangelidis,et al.  Ego-Motion Alignment from Face Detections for Collaborative Augmented Reality , 2020, ArXiv.

[27]  Neil Burgess,et al.  Coordinated hippocampal-entorhinal replay as structural inference , 2019, NeurIPS.

[28]  Byron Boots,et al.  Continuous-time Gaussian process motion planning via probabilistic inference , 2017, Int. J. Robotics Res..

[29]  William T. Freeman,et al.  Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology , 1999, Neural Computation.

[30]  Ian McGraw,et al.  Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing , 2006, UAI.

[31]  Stefano Ermon,et al.  Belief Propagation Neural Networks , 2020, NeurIPS.

[32]  Fu Jie Huang,et al.  A Tutorial on Energy-Based Learning , 2006 .

[33]  Yoshua Bengio,et al.  The Consciousness Prior , 2017, ArXiv.

[34]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[35]  Geoffrey E. Hinton,et al.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , 2017, ICLR.

[36]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[37]  Hai Jin,et al.  A Survey on Graph Processing Accelerators: Challenges and Opportunities , 2019, Journal of Computer Science and Technology.

[38]  Frank Arens,et al.  [Welcome to the Jungle]. , 2014, Pflege Zeitschrift.

[39]  William L. Briggs,et al.  A multigrid tutorial , 1987 .

[40]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[41]  Max Welling,et al.  Neural Enhanced Belief Propagation on Factor Graphs , 2020, AISTATS.

[42]  Noam Shazeer,et al.  Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , 2021, ArXiv.

[43]  Stefan Leutenegger,et al.  Bundle Adjustment on a Graph Processor , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[45]  Yik-Chung Wu,et al.  On Convergence Conditions of Gaussian Belief Propagation , 2015, IEEE Transactions on Signal Processing.

[46]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[47]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[48]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[49]  Soummya Kar,et al.  Convergence Analysis of Belief Propagation on Gaussian Graphical Models , 2018, ArXiv.

[50]  Wolfram Burgard,et al.  Robust map optimization using dynamic covariance scaling , 2013, 2013 IEEE International Conference on Robotics and Automation.

[51]  Matthew Cook,et al.  Factorized Computation: What the Neocortex Can Tell Us About the Future of Computing , 2018, Front. Comput. Neurosci..

[52]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .