Swift: Compiled Inference for Probabilistic Programming Languages

A probabilistic program defines a probability measure over its semantic structures. One common goal of probabilistic programming languages (PPLs) is to compute posterior probabilities for arbitrary models and queries, given observed evidence, using a generic inference engine. Most PPL inference engines---even the compiled ones---incur significant runtime interpretation overhead, especially for contingent and open-universe models. This paper describes Swift, a compiler for the BLOG PPL. Swift-generated code incorporates optimizations that eliminate interpretation overhead, maintain dynamic dependencies efficiently, and handle memory management for possible worlds of varying sizes. Experiments comparing Swift with other PPL engines on a variety of inference problems demonstrate speedups ranging from 12x to 326x.

[1]  Mark N. Wegman,et al.  An efficient method of computing static single assignment form , 1989, POPL '89.

[2]  Joseph Robert Horgan,et al.  Dynamic program slicing , 1990, PLDI '90.

[3]  Proceedings of the IEEE , 2018, IEEE Journal of Emerging and Selected Topics in Power Electronics.

[4]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[5]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[6]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[7]  Avi Pfeffer,et al.  IBAL: A Probabilistic Rational Programming Language , 2001, IJCAI.

[8]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[9]  Volume 25 , 2005 .

[10]  Stuart J. Russell,et al.  Approximate Inference for Infinite Contingent Bayesian Networks , 2005, AISTATS.

[11]  Stuart J. Russell,et al.  BLOG: Probabilistic Models with Unknown Objects , 2005, IJCAI.

[12]  Stuart J. Russell,et al.  General-Purpose MCMC Inference over Relational Structures , 2006, UAI.

[13]  Tom Minka,et al.  TrueSkillTM: A Bayesian Skill Rating System , 2006, NIPS.

[14]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[15]  David A. McAllester,et al.  Random-World Semantics and Syntactic Independence for Expressive Languages , 2008 .

[16]  A. Pfeffer,et al.  Figaro : An Object-Oriented Probabilistic Programming Language , 2009 .

[17]  Conrad Sanderson,et al.  Armadillo: An Open Source C++ Linear Algebra Library for Fast Prototyping and Computationally Intensive Experiments , 2010 .

[18]  Stuart J. Russell,et al.  Gibbs Sampling in Open-Universe Stochastic Languages , 2010, UAI.

[19]  Noah D. Goodman,et al.  Lightweight Implementations of Probabilistic Programming Languages Via Transformational Compilation , 2011, AISTATS.

[20]  W. Marsden.  I and J , 2012 .

[21]  Lei Li,et al.  Dynamic Scaled Sampling for Deterministic Constraints , 2013, AISTATS.

[22]  Joshua B. Tenenbaum,et al.  Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs , 2013, NIPS.

[23]  Sriram K. Rajamani,et al.  Efficiently Sampling Probabilistic Programs via Program Analysis , 2013, AISTATS.

[24]  Chung-Kil Hur,et al.  Slicing probabilistic programs , 2014, PLDI.

[25]  Frank D. Wood,et al.  A New Approach to Probabilistic Programming Inference , 2014, AISTATS.

[26]  Stuart J. Russell,et al.  BFiT: From Possible-World Semantics to Random-Evaluation Semantics in an Open Universe , 2014 .

[27]  Pat Hanrahan,et al.  Generating Efficient MCMC Kernels from Probabilistic Programs , 2014, AISTATS.

[28]  Chung-Kil Hur,et al.  R2: An Efficient MCMC Sampler for Probabilistic Programs , 2014, AAAI.

[29]  Michael F. P. O'Boyle,et al.  Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation , 2014, PLDI 2014.

[30]  Joseph Tassarotti,et al.  Augur: Data-Parallel Probabilistic Modeling , 2014, NIPS.

[31]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[32]  Andrew Gelman,et al.  Automatic Variational Inference in Stan , 2015, NIPS.

[33]  Noah D. Goodman,et al.  C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching , 2015, AISTATS.

[34]  Seyed Mehran Kazemi,et al.  Knowledge Compilation for Lifted Probabilistic Inference: Compiling to a Low-Level Language , 2016, KR.

[35]  Emina Torlak,et al.  SIMPL: A DSL for Automatic Specialization of Inference Algorithms , 2016, ArXiv.