Flexible debugging with controllable overhead

This thesis is concerned with the problem of discovering bugs in a program. This crucial part of debugging is becoming ever more relevant as the complexity of software systems increases. Symbolic debuggers retain their relevance for bug diagnosis once a bug has been isolated in a development environment; however, instrumentation continues to grow in relevance because it allows the programmer to specify execution points of interest in advance and let the program run normally. Instrumentation’s has a strong appeal if the nature of the bug is uncertain: the programmer can add checks or logging for a variety of interesting events and let the program run, automating what would otherwise be a painstaking manual process. Instrumentation is widely-used in development environments; nearly every large software project has benefited from verbose logging or some kind of execution profiling. In fact, recently DTrace has brought instrumentation to system administrators as well, allowing them to diagnose performance problems and errors on production servers. However, particularly with the proliferation of consumer devices running full Unix software stacks, it is becoming more and more useful to be able to apply the same techniques that were available in development and system-administration environments remotely, in end-user environments with minimal user impact. In this thesis, we present a body of work that addresses this desire. We have developed a GCC-based instrumentation system called adb that leverages the compiler’s intermediate representation, which includes type, control-flow, and dominator information, to enable instrumentation at a variety of locations, called probes, in a program. A developer can ship a program with deactivated probes, and later develop small consumers that use the data at these probes and insert them into deployed copies of the program dynamically, with no end-user intervention required. We have also developed a novel overhead-management policy called SMCO that ensures that instrumentation incurs fixed overhead. In addition to presenting adb and evaluating its performance guarantees, we also discuss a GCC plug-in system that we developed and that is being adopted by the GCC community for inclusion in release 4.5. Finally, we also describe overhead-control approaches we developed before SMCO, which put the design decisions made for adb in context.