Profile-Based, Load-Independent Anomaly Detection and Analysis in Performance Regression Testing of Software Systems

Performance evaluation through regression testing is an important step in the software production process. It aims to make sure that the performance of new releases do not regress under a field-like load. The main outputs of regression tests are the metrics that represent the response time of various transactions as well as the resource utilization (CPU, disk I/Oand Network). In this paper, we propose to use a concept known as Transaction Profile, which can provide a detailed representation for the transaction in a load independent manner, to detect anomalies through performance test runs. The approach uses data readily available in performance regression tests and a queueing network model of the system under test to infer the Transactions Profiles. Our initial results show that the Transactions Profiles calculated from load regression test data uncover the performance impact of any update to the software. Therefore we conclude that using Transactions Profiles is an effective approach to allow testing teams to easily assure each new software release does not suffer performance regression.

[1]  Zhen Ming Jiang,et al.  Automated analysis of load testing results , 2010, ISSTA '10.

[2]  John Murphy,et al.  Non-intrusive end-to-end runtime path tracing for J2EE systems , 2006, IEE Proc. Softw..

[3]  Ying Zou,et al.  Mining Performance Regression Testing Repositories for Automated Performance Analysis , 2010, 2010 10th International Conference on Quality Software.

[4]  Thanh H. D. Nguyen.  Using Control Charts for Detecting and Understanding Performance Regressions in Large Software , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[5]  Stephen Dawson,et al.  Estimating service resource consumption from response time measurements , 2009, VALUETOOLS.

[6]  Roberto Turrin,et al.  Robust Workload Estimation in Queueing Network Performance Models , 2008, 16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008).

[7]  Mel Ó Cinnéide,et al.  Improving Software Security Using Search-Based Refactoring , 2012, SSBSE.

[8]  Giuseppe Serazzi,et al.  JMT: performance engineering tools for system modeling , 2009, PERV.

[9]  Frank M. Bereznay.  Did something change? using statistical techniques to interpret service and resource metrics , 2006, Int. CMG Conference.

[10]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[11]  Evgenia Smirni,et al.  Analysis of application performance and its change via representative application signatures , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[12]  Leonid Grinshpan.  Solving Enterprise Applications Performance Puzzles: Queuing Models to the Rescue , 2012 .