Arthur : Rich Post-Facto Debugging for Production Analytics Applications

Debugging the massive parallel computations that run in today’s datacenters is hard, as they consist of thousands of tasks processing terabytes of data. It is especially hard in production settings, where performance overheads of more than a few percent are unacceptable. To address this challenge, we present Arthur, a new debugger that provides a rich set of analysis tools at close to zero runtime overhead through selective replay of data flow applications. Unlike previous replay debuggers, which add high overheads due to the need to log low-level nondeterministic events, Arthur takes advantage of the structure of large-scale data flow models (e.g., MapReduce), which split work into deterministic tasks for fault tolerance, to minimize its logging cost. We use selective replay to implement a variety of debugging features, including rerunning any task in a single-process debugger; ad-hoc queries on computation state; and forward and backward tracing of records through the computation, which we achieve using a program transformation at replay time. We implement Arthur for Hadoop and Spark, and show that it can be used to find a variety of real bugs.