Large-Scale Provenance for Soufflé

Logic programming languages, such as Datalog, have seen a rise in popularity in recent years, now being widely used to answer questions about real world problems. This popularity stems from the wide applicability of logic as a domain specific language for a variety of problems including static program analysis, declarative networking and security analysis. The use of logic reduces the complexity of implementing applications as programs are written in a declarative fashion. In other words, instead of describing computational steps imperatively, computations are concisely specified by their intended result. As a consequence, however, the lack of computational steps makes debugging very challenging, and there is no clear debugging strategy in logic programming. In the field of database research, provenance has been introduced as a way of explaining the output of relations, and can also be applied to Datalog. The big challenge with large real-world problems, however, is that the relations may contain billions of tuples, rendering existing provenance approaches infeasible. Hence, there is still a gap to design and implement scalable and reusable provenance systems for high-performance open-source Datalog engines, such as Soufflé. In this thesis, we develop the theory of provenance in the form of a proof tree for a given tuple. However, the construction of the proof tree in a naïve fashion is too expensive for large datasets. To ensure scalability, we push computation from logic evaluation time to proof construction time, achieving an efficient space/time trade-off. This lazy evaluation approach produces an annotated intensional database, which can be queried after evaluation by an arbitrary number of provenance queries without the need for recomputing the intensional database. We conduct experiments with complex industrialstrength benchmarks, including DOOP with DaCapo, which produce hundreds of millions of output tuples. We demonstrate that our novel provenance approach incurs a runtime and memory consumption overhead of 1.5× on average. Thus, it can cope with large datasets, where existing techniques and a naïve implementation become infeasible.