Church: a language for generative models

Formal languages for probabilistic modeling enable re-use, modularity, and descriptive clarity, and can foster generic inference techniques. We introduce Church, a universal language for describing stochastic generative processes. Church is based on the Lisp model of lambda calculus, containing a pure Lisp as its deterministic subset. The semantics of Church is defined in terms of evaluation histories and conditional distributions on such histories. Church also includes a novel language construct, the stochastic memoizer, which enables simple description of many complex non-parametric models. We illustrate language features through several examples, including: a generalized Bayes net in which parameters cluster over trials, infinite PCFGs, planning by inference, and various non-parametric clustering models. Finally, we show how to implement query on any Church program, exactly and approximately, using Monte Carlo techniques.

[1]  John McCarthy,et al.  A Basis for a Mathematical Theory of Computation , 1899 .

[2]  A. Church.  A Set of Postulates for the Foundation of Logic , 1932 .

[3]  John C. Reynolds,et al.  Definitional Interpreters for Higher-Order Programming Languages , 1972, ACM '72.

[4]  M.N. Sastry,et al.  Structure and interpretation of computer programs , 1986, Proceedings of the IEEE.

[5]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[6]  S. Muggleton.  Stochastic Logic Programs , 1996 .

[7]  Taisuke Sato,et al.  PRISM: A Language for Symbolic-Statistical Modeling , 1997, IJCAI.

[8]  R. Kent Dybvig,et al.  Revised5 Report on the Algorithmic Language Scheme , 1986, SIGP.

[9]  Robert Hieb,et al.  Revised 5 Report on the Algorithmic Language , 1999 .

[10]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[11]  Carl E. Rasmussen,et al.  Factorial Hidden Markov Models , 1997 .

[12]  Avi Pfeffer,et al.  IBAL: A Probabilistic Rational Programming Language , 2001, IJCAI.

[13]  Stuart J. Russell,et al.  BLOG: Probabilistic Models with Unknown Objects , 2005, IJCAI.

[14]  Stuart J. Russell,et al.  Probabilistic models with unknown objects , 2006 .

[15]  Thomas L. Griffiths,et al.  Learning Systems of Concepts with an Infinite Relational Model , 2006, AAAI.

[16]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[17]  J. Pitman.  Combinatorial Stochastic Processes , 2006 .

[18]  Marc Toussaint,et al.  Probabilistic inference for solving (PO) MDPs , 2006 .

[19]  Thomas L. Griffiths,et al.  Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models , 2006, NIPS.

[20]  Alexey Radul,et al.  Report on the probabilistic language scheme , 2007, DLS '07.

[21]  Dan Klein,et al.  The Infinite PCFG Using Hierarchical Dirichlet Processes , 2007, EMNLP.

[22]  Keith Bonawitz,et al.  Composable probabilistic inference with BLAISE , 2008 .

[23]  David A. McAllester,et al.  Random-World Semantics and Syntactic Independence for Expressive Languages , 2008 .