Automated root cause isolation of performance regressions during software development

Performance is crucial for the success of an application. To build responsive and cost efficient applications, software engineers must be able to detect and fix performance problems early in the development process. Existing approaches are either relying on a high level of abstraction such that critical problems cannot be detected or require high manual effort. In this paper, we present a novel approach that integrates performance regression root cause analysis into the existing development infrastructure using performance-aware unit tests and the revision history. Our approach is easy to use and provides software engineers immediate insights with automated root cause analysis. In a realistic case study based on the change history of Apache Commons Math, we demonstrate that our approach can automatically detect and identify the root cause of a major performance regression.

[1]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[2]  Ahmed E. Hassan,et al.  A qualitative study on performance bugs , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[3]  Wilhelm Hasselbring,et al.  Kieker: continuous monitoring and on demand visualization of Java software behavior , 2008, ICSE 2008.

[4]  Matthias Hauswirth,et al.  Catch me if you can: performance bug detection in the wild , 2011, OOPSLA '11.

[5]  Andreas Zeller,et al.  Generating parameterized unit tests , 2011, ISSTA '11.

[6]  Chen Fu,et al.  Automatically finding performance problems with feedback-directed learning software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[7]  Petr Tuma,et al.  Capturing performance assumptions using stochastic performance logic , 2012, ICPE '12.

[8]  Chandra Krintz,et al.  Tracking performance across software revisions , 2009, PPPJ '09.

[9]  Steffen Becker,et al.  Predicting the Performance of Component-Based Software Architectures with Different Usage Profiles , 2007, QoSA.

[10]  E. Zimran,et al.  Performance engineering throughout the product life cycle , 1993, 1993 CompEuro Proceedings Computers in Design, Manufacturing, and Production.

[11]  Muga Nishizawa,et al.  An Easy-to-Use Toolkit for Efficient Java Bytecode Translators , 2003, GPCE.

[12]  Allen D. Malony,et al.  Integrating Performance Analysis in the Uintah Software Development Cycle , 2009, ISHPC.

[13]  Atanas Rountev,et al.  Uncovering performance problems in Java applications with reference propagation profiling , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[14]  Ahmed E. Hassan,et al.  Security versus performance bugs: a case study on Firefox , 2011, MSR '11.

[15]  Barton P. Miller,et al.  Diagnosing Distributed Systems with Self-propelled Instrumentation , 2008, Middleware.

[16]  Wilhelm Hasselbring,et al.  Self-adaptive software system monitoring for performance anomaly localization , 2011, ICAC '11.

[17]  Daniel A. Menascé Software, performance, or engineering? , 2002, WOSP '02.

[18]  Donghun Lee,et al.  A Performance Anomaly Detection and Analysis Framework for DBMS Development , 2012, IEEE Transactions on Knowledge and Data Engineering.

[19]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[20]  John Murphy,et al.  Detecting Performance Antipatterns in Component Based Enterprise Systems , 2008, J. Object Technol..

[21]  Allen D. Malony,et al.  Performance Analysis Integration in the Uintah Software Development Cycle , 2003, International Journal of Parallel Programming.

[22]  Lieven Eeckhout,et al.  Statistically rigorous java performance evaluation , 2007, OOPSLA.

[23]  Gilbert Hamann,et al.  Automated performance analysis of load tests , 2009, 2009 IEEE International Conference on Software Maintenance.

[24]  Ying Zou,et al.  Mining Performance Regression Testing Repositories for Automated Performance Analysis , 2010, 2010 10th International Conference on Quality Software.

[25]  Barton P. Miller,et al.  The Paradyn Parallel Performance Measurement Tool , 1995, Computer.

[26]  Petr Tuma,et al.  Repeated results analysis for middleware regression benchmarking , 2005, Perform. Evaluation.