Trace types and denotational semantics for sound programmable inference in probabilistic languages

Modern probabilistic programming languages aim to formalize and automate key aspects of probabilistic modeling and inference. Many languages provide constructs for programmable inference that enable developers to improve inference speed and accuracy by tailoring an algorithm for use with a particular model or dataset. Unfortunately, it is easy to use these constructs to write unsound programs that appear to run correctly but produce incorrect results. To address this problem, we present a denotational semantics for programmable inference in higher-order probabilistic programming languages, along with a type system that ensures that well-typed inference programs are sound by construction. A central insight is that the type of a probabilistic expression can track the space of its possible execution traces, not just the type of value that it returns, as these traces are often the objects that inference algorithms manipulate. We use our semantics and type system to establish soundness properties of custom inference programs that use constructs for variational, sequential Monte Carlo, importance sampling, and Markov chain Monte Carlo inference.

[1]  Hongseok Yang,et al.  Towards verified stochastic variational inference for probabilistic programs , 2019, Proc. ACM Program. Lang..

[2]  Vikash K. Mansinghka,et al.  Gen: a general-purpose probabilistic programming system with programmable inference , 2019, PLDI.

[3]  Hugo Paquet,et al.  Probabilistic Programming Inference via Intensional Semantics , 2019, ESOP.

[4]  Frank D. Wood,et al.  LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models , 2019, AISTATS.

[5]  Martin C. Rinard,et al.  Bayesian synthesis of probabilistic programs for automatic data modeling , 2019, Proc. ACM Program. Lang..

[6]  Noah D. Goodman,et al.  Pyro: Deep Universal Probabilistic Programming , 2018, J. Mach. Learn. Res..

[7]  Ohad Kammar,et al.  Functional programming for modular Bayesian inference , 2018, Proc. ACM Program. Lang..

[8]  Gilles Barthe,et al.  Formal verification of higher-order probabilistic programs: reasoning about approximation, convergence, Bayesian inference, and optimization , 2018, Proc. ACM Program. Lang..

[9]  Vikash K. Mansinghka,et al.  A design proposal for Gen: probabilistic programming with fast custom inference via code generation , 2018, MAPL@PLDI.

[10]  Vikash K. Mansinghka,et al.  Probabilistic programming with programmable inference , 2018, PLDI.

[11]  Eric Atkinson,et al.  Verifying Handcoded Probabilistic Inference Procedures , 2018, ArXiv.

[12]  Hedvig Kjellström,et al.  Advances in Variational Inference , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Ohad Kammar,et al.  Denotational validation of higher-order Bayesian inference , 2017, Proc. ACM Program. Lang..

[14]  J. Gregory Morrisett,et al.  Compiling Markov chain Monte Carlo algorithms for probabilistic modeling , 2017, PLDI.

[15]  Vikash K. Mansinghka,et al.  AIDE: An algorithm for measuring the accuracy of probabilistic inference algorithms , 2017, NIPS.

[16]  Dustin Tran,et al.  Deep Probabilistic Programming , 2017, ICLR.

[17]  Marcus A. Brubaker,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[18]  Ohad Kammar,et al.  A convenient category for higher-order probability theory , 2017, 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[19]  Noah D. Goodman,et al.  Deep Amortized Inference for Probabilistic Programs , 2016, ArXiv.

[20]  Shin-ya Katsumata,et al.  Towards a Formal Theory of Graded Monads , 2016, FoSSaCS.

[21]  Chung-chieh Shan,et al.  Composing Inference Algorithms as Program Transformations , 2016, UAI.

[22]  Jacques Carette,et al.  Probabilistic Inference by Program Transformation in Hakaru (System Description) , 2016, FLOPS.

[23]  Ohad Kammar,et al.  Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints , 2016, 2016 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[24]  P. Diaconis,et al.  The sample size required in importance sampling , 2015, 1511.01437.

[25]  Zoubin Ghahramani,et al.  Practical probabilistic programming with monads , 2015, Haskell.

[26]  Yura N. Perov,et al.  Venture: a higher-order probabilistic programming platform with programmable inference , 2014, ArXiv.

[27]  Frank D. Wood,et al.  A New Approach to Probabilistic Programming Inference , 2014, AISTATS.

[28]  Shin-ya Katsumata,et al.  Parametric effect monads and semantics of effect systems , 2014, POPL.

[29]  Lawrence M. Murray.  Bayesian State-Space Modelling on High-Performance Hardware Using LibBi , 2013, 1306.3277.

[30]  David Wingate,et al.  Automated Variational Inference in Probabilistic Programming , 2013, ArXiv.

[31]  Alexander G. Gray,et al.  A type theory for probability density functions , 2012, POPL '12.

[32]  Noah D. Goodman,et al.  Lightweight Implementations of Probabilistic Programming Languages Via Transformational Compilation , 2011, AISTATS.

[33]  Yishay Mansour,et al.  Learning Bounds for Importance Weighting , 2010, NIPS.

[34]  Noam M. Shazeer,et al.  Variational Program Inference , 2010, ArXiv.

[35]  Persi Diaconis,et al.  The Markov chain Monte Carlo revolution , 2008 .

[36]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[37]  Joseph T. Chang,et al.  Conditioning as disintegration , 1997 .

[38]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[39]  Artificial Intelligence A Modern Approach 3rd , 2022 .

[40]  Marcus Gallagher,et al.  Reversible Jump Probabilistic Programming , 2019, AISTATS.

[41]  Adam Ścibior,et al.  Strongly typed tracing of probabilistic programs , 2019 .

[42]  Zoubin Ghahramani,et al.  Turing: A Language for Flexible Probabilistic Inference , 2018 .

[43]  Hongseok Yang,et al.  The semantic structure of quasi-Borel spaces , 2018 .

[44]  Noah D. Goodman,et al.  Variational Optimal Experiment Design : Efficient Automation of Adaptive Experiments , 2018 .

[45]  David B. Dunson,et al.  Bayesian data analysis, third edition , 2013 .

[46]  Daniel M. Roy.  When are probabilistic programs probably computationally tractable? , 2010 .

[47]  Keith Bonawitz,et al.  Composable probabilistic inference with BLAISE , 2008 .

[48]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[49]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[50]  Jun S. Liu,et al.  Sequential Monte Carlo methods for dynamic systems , 1997 .