Declarative Information Extraction in a Probabilistic Database System

Full-text documents represent a large fraction of the world’s data. Although not structured per se, they often contain snippets of structured information within them: e.g., names, addresses, and document titles. Information Extraction (IE) techniques identify such structured information in text. In recent years, database research has pursued IE on two fronts: declarative languages and systems for managing IE tasks, and IE as an uncertain data source for Probabilistic Databases. It is natural to consider merging these two directions, but efforts to do so have had to compromise on the statistical robustness of IE algorithms in order to fit with early Probabilistic Database models. In this paper, we bridge the gap between these ideas by implementing a state-of-the-art statistical IE approach – Conditional Random Fields (CRFs) – in the setting of Probabilistic Databases that treat statistical models as first-class data objects. Using standard relational tables to capture CRF parameters, and inverted-file representations of text, we show that the Viterbi algorithm for CRF inference can be specified declaratively in recursive SQL, in a manner that can both choose likely segmentations, and provide detailed marginal distributions for label assignment. Given this implementation, we propose query processing optimizations that effectively combine probabilistic inference and relational operators such as selections and joins. In an experimental study with two data sets, we demonstrate the efficiency of our in-database Viterbi implementation in PostgreSQL relative to an open-source CRF library, and show the performance benefits of our optimizations.