Compiling Stan to generative probabilistic languages and extension to deep probabilistic programming

Stan is a probabilistic programming language that is popular in the statistics community, with a high-level syntax for expressing probabilistic models. Stan differs by nature from generative probabilistic programming languages like Church, Anglican, or Pyro. This paper presents a comprehensive compilation scheme to compile any Stan model to a generative language and proves its correctness. We use our compilation scheme to build two new backends for the Stanc3 compiler targeting Pyro and NumPyro. Experimental results show that the NumPyro backend yields a 2.3x speedup compared to Stan in geometric mean over 26 benchmarks. Building on Pyro we extend Stan with support for explicit variational inference guides and deep probabilistic models. That way, users familiar with Stan get access to new features without having to learn a fundamentally new language.

[1]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[2]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[3]  Robert E Weiss,et al.  Bayesian methods for data analysis. , 2010, American journal of ophthalmology.

[4]  G. Andrew,et al.  arm: Data Analysis Using Regression and Multilevel/Hierarchical Models , 2014 .

[5]  Vikash K. Mansinghka,et al.  Gen: a general-purpose probabilistic programming system with programmable inference , 2019, PLDI.

[6]  Sam Staton,et al.  Commutative Semantics for Probabilistic Programming , 2017, ESOP.

[7]  Hongseok Yang,et al.  An Introduction to Probabilistic Programming , 2018, ArXiv.

[8]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[9]  John Salvatier,et al.  Probabilistic programming in Python using PyMC3 , 2016, PeerJ Comput. Sci..

[10]  Frank D. Wood,et al.  Learning Disentangled Representations with Semi-Supervised Deep Generative Models , 2017, NIPS.

[11]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[12]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[13]  Jun Zhu,et al.  ZhuSuan: A Library for Bayesian Deep Learning , 2017, ArXiv.

[14]  Martin Hirzel,et al.  Replication package for the article: Compiling Stan to Generative Probabilistic Languages and Extension to Deep Probabilistic Programming , 2021, Artifact Digital Object Group.

[15]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[16]  Dexter Kozen,et al.  Semantics of probabilistic programs , 1979, 20th Annual Symposium on Foundations of Computer Science (sfcs 1979).

[17]  Thomas A. Henzinger,et al.  Probabilistic programming , 2014, FOSE.

[18]  Noah D. Goodman,et al.  Pyro: Deep Universal Probabilistic Programming , 2018, J. Mach. Learn. Res..

[19]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[20]  Hongseok Yang,et al.  Towards verified stochastic variational inference for probabilistic programs , 2019, Proc. ACM Program. Lang..

[21]  Neeraj Pradhan,et al.  Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro , 2019, ArXiv.

[22]  Dustin Tran,et al.  Deep Probabilistic Programming , 2017, ICLR.

[23]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[24]  Andrew D. Gordon,et al.  Probabilistic programming with densities in SlicStan: efficient, flexible, and deterministic , 2018, Proc. ACM Program. Lang..

[25]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..

[26]  Andrew Thomas,et al.  The BUGS project: Evolution, critique and future directions , 2009, Statistics in medicine.

[27]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[28]  Martin Hirzel,et al.  Deep Probabilistic Programming Languages: A Qualitative Study , 2018, ArXiv.

[29]  David Tolpin,et al.  Design and Implementation of Probabilistic Programming Language Anglican , 2016, IFL 2016.

[30]  Dan A. Simovici,et al.  Bayesian Learning , 2019, Variational Bayesian Learning Theory.

[31]  Ohad Kammar,et al.  Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints , 2016, 2016 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[32]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .