BorderPatrol: Isolating Events for Precise Black-box Tracing

High-level causal request traces are of interest to developers of large concurrent and distributed applications. These traces show how a request is processed as it passes through several modules which may be processes, threads, machines, or devices. They aid programmer understanding and are increasingly analyzed by tools used to detect performance and correctness errors. Precise traces are more useful than statistical approaches because they can detect anomalous behavior and allow decisions at run-time. Since these traces are difficult to obtain without application-specific instrumentation of each module of the system, much of the recent work that analyzes request traces is limited to applications for which source code and developer expertise is available. We present BorderPatrol, which obtains precise request traces through systems built from a litany of unmodified modules, written in varied languages, with varying architectures. These include Apache, thttpd, PostgreSQL, TurboGears, BIND and notably Zeus, a closed-source event-driven HTTP/1.1 web server, which uses helper processes. BorderPatrol obtains these traces using active observation which slightly modifies the event stream observed by system modules, simplifying precise observation. Protocol processors aid active observation by leveraging knowledge about standard protocols and interfaces between concurrent modules, avoiding the need for implementation-specific instrumentation. BorderPatrol obtains precise traces for black-box systems that cannot be traced by any other technique. Further, it does so with limited overhead on real systems (approximately 10-15%) making it a viable option for deployment on produc-