Automatic, load-independent detection of performance regressions by transaction profiles

Performance regression testing is an important step in the production process of enterprise applications. Yet, analysing this type of testing data is mainly conducted manually and depends on the load applied during the test. To ease such a manual task we present an automated, load-independent technique to detect performance regression anomalies based on the analysis of performance testing data using a concept known as Transaction Profile. The approach can be automated and it utilises data already available to the performance testing along with the queueing network model of the testing system. The presented ``Transaction Profile Run Report'' was able to automatically catch performance regression anomalies ca-used by software changes and isolate them from those caused by load variations with a precision of 80% in a case study conducted against an open source application. Hence, by deploying our system, the testing teams are able to detect performance regression anomalies by avoiding the manual approach and eliminating the need to do extra runs with varying load.

[1]  Frank M. Bereznay.  Did something change? using statistical techniques to interpret service and resource metrics , 2006, Int. CMG Conference.

[2]  Leonid Grinshpan.  Solving Enterprise Applications Performance Puzzles: Queuing Models to the Rescue , 2012 .

[3]  Thanh H. D. Nguyen.  Using Control Charts for Detecting and Understanding Performance Regressions in Large Software , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[4]  Virgílio A. F. Almeida,et al.  Performance by Design - Computer Capacity Planning By Example , 2004 .

[5]  J. Ben Atkinson,et al.  An Introduction to Queueing Networks , 1988 .

[6]  Scott F. Midkiff,et al.  Queueing network analysis: concepts, terminology, and methods , 2003, J. Syst. Softw..

[7]  Edward D. Lazowska,et al.  Quantitative system performance - computer system analysis using queueing network models , 1983, Int. CMG Conference.

[8]  Roberto Turrin,et al.  Robust Workload Estimation in Queueing Network Performance Models , 2008, 16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008).

[9]  Virgílio A. F. Almeida,et al.  Capacity Planning for Web Services: Metrics, Models, and Methods , 2001 .

[10]  Jean C. Walrand,et al.  An introduction to queueing networks , 1989, Prentice Hall International editions.

[11]  Giuseppe Serazzi.  Performance Evaluation Modelling with JMT: learning by examples , 2008 .

[12]  C. Murray Woodside,et al.  Enhanced Modeling and Solution of Layered Queueing Networks , 2009, IEEE Transactions on Software Engineering.

[13]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[14]  Miao Wang,et al.  Profile-Based, Load-Independent Anomaly Detection and Analysis in Performance Regression Testing of Software Systems , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[15]  Zhen Ming Jiang,et al.  Automated analysis of load testing results , 2010, ISSTA '10.

[16]  Edward D. Lazowska,et al.  Quantitative System Performance , 1985, Int. CMG Conference.

[17]  Giuseppe Serazzi,et al.  Java Modelling Tools: an Open Source Suite for Queueing Network Modelling andWorkload Analysis , 2006, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[18]  K. Mani Chandy,et al.  Open, Closed, and Mixed Networks of Queues with Different Classes of Customers , 1975, JACM.

[19]  Yujia Ge,et al.  A Reactivity-based Framework of Automated Performance Testing for Web Applications , 2010, 2010 Ninth International Symposium on Distributed Computing and Applications to Business, Engineering and Science.

[20]  Ray Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.