FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs

Discriminatively trained undirected graphical models have had wide empirical success, and there has been increasing interest in toolkits that ease their application to complex relational data. The power in relational models is in their repeated structure and tied parameters; at issue is how to define these structures in a powerful and flexible way. Rather than using a declarative language, such as SQL or first-order logic, we advocate using an imperative language to express various aspects of model structure, inference, and learning. By combining the traditional, declarative, statistical semantics of factor graphs with imperative definitions of their construction and operation, we allow the user to mix declarative and procedural domain knowledge, and also gain significant efficiencies. We have implemented such imperatively defined factor graphs in a system we call FACTORIE, a software library for an object-oriented, strongly-typed, functional language. In experimental comparisons to Markov Logic Networks on joint segmentation and coreference, we find our approach to be 3-15 times faster while reducing error by 20-25%—achieving a new state of the art.

[1]  Andrew McCallum,et al.  Tractable Learning and Inference with High-Order Representations , 2006 .

[2]  Andrew McCallum,et al.  A Machine Learning Approach to Building Domain-Specific Search Engines , 1999, IJCAI.

[3]  Martin Odersky,et al.  An Overview of the Scala Programming Language Second Edition , 2006 .

[4]  Pedro M. Domingos,et al.  A General Method for Reducing the Complexity of Relational Inference and its Application to MCMC , 2008, AAAI.

[5]  Luc De Raedt,et al.  Probabilistic logic learning , 2003, SKDD.

[6]  A. Pfeffer,et al.  Figaro : An Object-Oriented Probabilistic Programming Language , 2009 .

[7]  Ben Taskar,et al.  Discriminative Probabilistic Models for Relational Data , 2002, UAI.

[8]  Andrew McCallum,et al.  Learning and inference in weighted logic with application to natural language processing , 2008 .

[9]  Ben Taskar,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[10]  Andrew McCallum,et al.  Collective Segmentation and Labeling of Distant Entities in Information Extraction , 2004 .

[11]  Jude W. Shavlik,et al.  Creating Advice-Taking Reinforcement Learners , 1998, Machine Learning.

[12]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[13]  Andrew McCallum,et al.  An Integrated, Conditional Model of Information Extraction and Coreference with Appli , 2004, UAI.

[14]  Dan Roth,et al.  The Necessity of Syntactic Parsing for Semantic Role Labeling , 2005, IJCAI.

[15]  Pedro M. Domingos,et al.  Joint Inference in Information Extraction , 2007, AAAI.

[16]  Keith Bonawitz,et al.  Composable probabilistic inference with BLAISE , 2008 .

[17]  Michael L. Wick,et al.  Inference and Learning in Large Factor Graphs with Adaptive Proposal Distributions and a Rank-based Objective , 2009 .

[18]  Avi Pfeffer,et al.  IBAL: A Probabilistic Rational Programming Language , 2001, IJCAI.

[19]  Koby Crammer,et al.  Global Discriminative Learning for Higher-Accuracy Computational Gene Prediction , 2007, PLoS Comput. Biol..

[20]  Taisuke Sato,et al.  PRISM: A Language for Symbolic-Statistical Modeling , 1997, IJCAI.

[21]  Luc De Raedt,et al.  Inductive Logic Programming: Theory and Methods , 1994, J. Log. Program..

[22]  Andrew McCallum,et al.  Bi-directional Joint Inference for Entity Resolution and Segmentation Using Imperatively-Defined Factor Graphs , 2009, ECML/PKDD.

[23]  Stuart J. Russell,et al.  Probabilistic models with unknown objects , 2006 .

[24]  Trevor Darrell,et al.  Conditional Random Fields for Object Recognition , 2004, NIPS.

[25]  Andrew McCallum,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[26]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[27]  Andrew McCallum,et al.  First-Order Probabilistic Models for Coreference Resolution , 2007, NAACL.

[28]  David Poole,et al.  Probabilistic Horn Abduction and Bayesian Networks , 1993, Artif. Intell..

[29]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.