Saul: Towards Declarative Learning Based Programming

We present Saul, a new probabilistic programming language designed to address some of the shortcomings of programming languages that aim at advancing and simplifying the development of AI systems. Such languages need to interact with messy, naturally occurring data, to allow a programmer to specify what needs to be done at an appropriate level of abstraction rather than at the data level, to be developed on a solid theory that supports moving to and reasoning at this level of abstraction and, finally, to support flexible integration of these learning and inference models within an application program. Saul is an object-functional programming language written in Scala that facilitates these by (1) allowing a programmer to learn, name and manipulate named abstractions over relational data; (2) supporting seamless incorporation of trainable (probabilistic or discriminative) components into the program, and (3) providing a level of inference over trainable models to support composition and make decisions that respect domain and application constraints. Saul is developed over a declaratively defined relational data model, can use piecewise learned factor graphs with declaratively specified learning and inference objectives, and it supports inference over probabilistic models augmented with declarative knowledge-based constraints. We describe the key constructs of Saul and exemplify its use in developing applications that require relational feature engineering and structured output prediction.

[1]  Yura N. Perov,et al.  Venture: a higher-order probabilistic programming platform with programmable inference , 2014, ArXiv.

[2]  Carina Silberer,et al.  Proceedings of the International Conference on Language Resources and Evaluation (LREC) , 2008 .

[3]  Dan Roth,et al.  Integer linear programming inference for conditional random fields , 2005, ICML.

[4]  Ming-Wei Chang,et al.  Unified Expectation Maximization , 2012, NAACL.

[5]  Andrew McCallum,et al.  FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs , 2009, NIPS.

[6]  Martin Odersky,et al.  Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition , 2010 .

[7]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[8]  Torres Martins,et al.  The Geometry of Constrained Structured Prediction: Applications to Inference and Learning of Natural Language Syntax , 2012 .

[9]  Dan Roth,et al.  Learning and Inference over Constrained Output , 2005, IJCAI.

[10]  Tommi S. Jaakkola,et al.  Approximate inference in graphical models using lp relaxations , 2010 .

[11]  Rina Dechter,et al.  Mixed deterministic and probabilistic networks , 2008, Annals of Mathematics and Artificial Intelligence.

[12]  David Maxwell Chickering,et al.  Dependency Networks for Inference, Collaborative Filtering, and Data Visualization , 2000, J. Mach. Learn. Res..

[13]  Johann Schumann,et al.  Under Consideration for Publication in J. Functional Programming Autobayes: a System for Generating Data Analysis Programs from Statistical Models , 2022 .

[14]  Luc De Raedt,et al.  Probabilistic logic learning , 2003, SKDD.

[15]  A. Pfeffer,et al.  Figaro : An Object-Oriented Probabilistic Programming Language , 2009 .

[16]  Frank Dellaert,et al.  DDF-SAM: Fully distributed SLAM using Constrained Factor Graphs , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Pedro M. Domingos.  1 Markov Logic: A Unifying Framework for Statistical Relational Learning , 2010 .

[18]  Dan Roth,et al.  On Kernel Methods for Relational Learning , 2003, ICML.

[19]  Ben Taskar,et al.  Posterior Regularization for Structured Latent Variable Models , 2010, J. Mach. Learn. Res..

[20]  Dan Roth.  Learning Based Programming , 1999 .

[21]  Ben Taskar,et al.  Markov Logic: A Unifying Framework for Statistical Relational Learning , 2007 .

[22]  Dan Roth,et al.  Learning Based Java for Rapid Development of NLP Systems , 2010, LREC.

[23]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[24]  Andrew McCallum,et al.  Piecewise training for structured prediction , 2009, Machine Learning.

[25]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[26]  Walter R. Gilks,et al.  A Language and Program for Complex Bayesian Modelling , 1994 .

[27]  Dan Roth,et al.  Efficient Decomposed Learning for Structured Prediction , 2012, ICML.

[28]  Ben Taskar,et al.  Discriminative Probabilistic Models for Relational Data , 2002, UAI.

[29]  Luc De Raedt,et al.  kLog: A Language for Logical and Relational Learning with Kernels (Extended Abstract) , 2012, IJCAI.

[30]  J. Besag.  Efficiency of pseudolikelihood estimation for simple Gaussian fields , 1977 .

[31]  Andrew McCallum,et al.  Piecewise Training for Undirected Models , 2005, UAI.

[32]  L. Getoor,et al.  1 Global Inference for Entity and Relation Identification via a Linear Programming Formulation , 2007 .

[33]  D. Heckerman,et al.  Dependency networks for inference , 2000 .

[34]  Taisuke Sato,et al.  PRISM: A Language for Symbolic-Statistical Modeling , 1997, IJCAI.

[35]  Shan Shan Huang,et al.  Datalog and emerging applications: an interactive tutorial , 2011, SIGMOD '11.

[36]  Ming-Wei Chang,et al.  Guiding Semi-Supervision with Constraint-Driven Learning , 2007, ACL.

[37]  Ming-Wei Chang,et al.  Structured learning with constrained conditional models , 2012, Machine Learning.

[38]  Esteban Zimányi,et al.  An Analytics-Aware Conceptual Model for Evolving Graphs , 2013, DaWaK.

[39]  Michael Collins,et al.  Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms , 2002, EMNLP.

[40]  D. Roth.  1 Global Inference for Entity and Relation Identification via a Linear Programming Formulation , 2007 .