Generating Efficient MCMC Kernels from Probabilistic Programs

Universal probabilistic programming languages (such as Church [6]) trade performance for abstraction: any model can be represented compactly as an arbitrary stochastic computation, but costly online analyses are required for inference. We present a technique that recovers hand-coded levels of performance from a universal probabilistic language, for the Metropolis-Hastings (MH) MCMC inference algorithm. It takes a Church program as input and traces its execution to remove computation overhead. It then analyzes the trace for each proposal, using slicing, to identify the minimal computation needed to evaluate the MH acceptance probability. Generated incremental code is much faster than a baseline implementation (up to 600x) and usually as fast as handcoded MH kernels.

[1]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[2]  Jan Vitek,et al.  Terra: a multi-stage language for high-performance computing , 2013, PLDI.

[3]  Noah D. Goodman,et al.  Lightweight Implementations of Probabilistic Programming Languages Via Transformational Compilation , 2011, AISTATS.

[4]  Andrew McCallum,et al.  FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs , 2009, NIPS.

[5]  Robert Harper.  Self-adjusting computation , 2004, LICS 2004.

[6]  John Aycock,et al.  A brief history of just-in-time , 2003, CSUR.

[7]  Stuart J. Russell,et al.  BLOG: Probabilistic Models with Unknown Objects , 2005, IJCAI.

[8]  Mason Chang,et al.  Trace-based just-in-time type specialization for dynamic languages , 2009, PLDI '09.

[9]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[10]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools (2nd Edition) , 2006 .

[11]  A. Pfeffer,et al.  Figaro : An Object-Oriented Probabilistic Programming Language , 2009 .

[12]  Noah D. Goodman,et al.  A Dynamic Programming Algorithm for Inference in Recursive Probabilistic Programs , 2012, StarAI@UAI.

[13]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[14]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[15]  Norman Ramsey,et al.  Stochastic lambda calculus and monads of probability distributions , 2002, POPL '02.

[16]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[17]  Pat Hanrahan,et al.  Synthesizing open worlds with constraints using locally annealed reversible jump MCMC , 2012, ACM Trans. Graph..

[18]  Daniel M. Roy,et al.  Posterior distributions are computable from predictive distributions , 2010, AISTATS.

[19]  Avi Pfeffer,et al.  IBAL: A Probabilistic Rational Programming Language , 2001, IJCAI.

[20]  Robert Harper,et al.  Self-adjusting computation , 2004, Proceedings of the 19th Annual IEEE Symposium on Logic in Computer Science, 2004..

[21]  Chung-chieh Shan,et al.  Embedded Probabilistic Programming , 2009, DSL.

[22]  Michael Franz,et al.  HotpathVM: an effective JIT compiler for resource-constrained devices , 2006, VEE '06.