Automatic Root-cause Diagnosis of Performance Anomalies in Production Software

Troubleshooting the performance of complex production sof tware is challenging. Most existing tools, such as profiling, trac ing, and logging systems, reveal what events occurred during performance anomalies. However, the users of such tools must then infer why these events occurred during a particular execution; e.g., that their execution was due to a specific input request or configuration setting. Because manual root cause determination is time-cons umi g and difficult, this paper introduces performance summarization , a technique for automatically inferring the root cause of per ormance problems. Performance summarization first attributes perf ormance costs to fine-grained events such as individual instruction s and system calls. It then uses dynamic information flow to determine the probable root causes for the execution of each event. The cos t of each event is assigned to root causes according to the relati ve probability that the causes led to the execution of that event. Fi nally, the total cost for each root cause is calculated by summing th e percause costs of all events. This paper also describes a differ ential form of performance summarization that compares two activi ties. We have implemented a tool called X-ray that performs perfor mance summarization. Our experimental results show that Xray accurately diagnoses 14 performance issues in the Apache HT TP server, Postfix mail server and PostgreSQL database, while a dding only 1–7% overhead to production systems.