Autonomic load-testing framework

In this paper, we present a method for performance testing of transactional systems. The methods models the system under test, finds the software and hardware bottlenecks and generate the workloads that saturate them. The framework is autonomic, the model and workloads are determined during the performance test execution by measuring the system performance, fitting a performance model and by analytically computing the number and mix of users that will saturate the bottlenecks. We model the software system using a two-layer queuing model and use analytical techniques to find the workload mixes that change the bottlenecks in the system. Those workload mixes become stress vectors and initial starting points for the stress test cases. The rest of test cases are generated based on a feedback loop that drives the software system towards the worst case behaviour.

[1]  Marin Litoiu,et al.  Hierarchical Model-based Autonomic Control of Software Systems , 2005 .

[2]  Daniel A. Menascé,et al.  Performance Engineering of Component-Based Distributed Software Systems , 2001, Performance Engineering.

[3]  Hassan Charaf,et al.  Modeling the Effect of Application Server Settings on the Performance of J2EE Web Applications , 2006, TEAA.

[4]  Marin Litoiu,et al.  Tracking time-varying parameters in software systems with extended Kalman filters , 2015, CASCON.

[5]  Marin Litoiu,et al.  Designing Process Replication and Activation: A Quantitative Approach , 2000, IEEE Trans. Software Eng..

[6]  Neil J. Gunther.  Guerrilla capacity planning - a tactical approach to planning for highly scalable applications and services , 2006 .

[7]  Daniel A. Menascé,et al.  Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning , 2000 .

[8]  Stephen S. Lavenberg,et al.  Mean-Value Analysis of Closed Multichain Queuing Networks , 1980, JACM.

[9]  Virgílio A. F. Almeida,et al.  Capacity Planning for Web Performance: Metrics, Models, and Methods , 1998 .

[10]  Marin Litoiu,et al.  The use of optimal filters to track parameters of performance models , 2005, Second International Conference on the Quantitative Evaluation of Systems (QEST'05).

[11]  Simona Bernardi,et al.  Performance aware open-world software in a 3-layer architecture , 2010, WOSP/SIPEW '10.

[12]  Derek L. Eager,et al.  Performance bound hierarchies for queueing networks , 1982, TOCS.

[13]  Daniel A. Menascé,et al.  Simple analytic modeling of software contention , 2002, PERV.

[14]  Gilbert Hamann,et al.  Automatic identification of load testing problems , 2008, 2008 IEEE International Conference on Software Maintenance.

[15]  Marin Litoiu,et al.  Tracking adaptive performance models using dynamic clustering of user classes , 2011, ICPE '11.

[16]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[17]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[18]  Daniel A. Menascé,et al.  QoS management in service-oriented architectures , 2007, Perform. Evaluation.

[19]  Edward D. Lazowska,et al.  Quantitative system performance - computer system analysis using queueing network models , 1983, Int. CMG Conference.

[20]  Gilbert Hamann,et al.  Automated performance analysis of load tests , 2009, 2009 IEEE International Conference on Software Maintenance.

[21]  Ying Zou,et al.  Mining Performance Regression Testing Repositories for Automated Performance Analysis , 2010, 2010 10th International Conference on Quality Software.

[22]  Dharmesh Thakkar,et al.  AUTOMATED CAPACITY PLANNING AND SUPPORT FOR ENTERPRISE APPLICATIONS , 2009 .

[23]  Katta G. Murty,et al.  Nonlinear Programming Theory and Algorithms , 2007, Technometrics.

[24]  Gilbert Hamann,et al.  A framework for measurement based performance modeling , 2008, WOSP '08.

[25]  Yixin Diao,et al.  Feedback Control of Computing Systems , 2004 .

[26]  Lui Sha,et al.  Feedback control with queueing-theoretic prediction for relative delay guarantees in web servers , 2003, The 9th IEEE Real-Time and Embedded Technology and Applications Symposium, 2003. Proceedings..

[27]  John Zahorjan,et al.  Balanced job bound analysis of queueing networks , 1982, CACM.

[28]  Gilbert Hamann,et al.  Using Load Tests to Automatically Compare the Subsystems of a Large Enterprise System , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[29]  Giuseppe Serazzi,et al.  Asymptotic Analysis of Multiclass Closed Queueing Networks: Multiple Bottlenecks , 1997, Perform. Evaluation.

[30]  Agnes Bogardi-Meszoly,et al.  Improved performance models of web-based software systems , 2009, 2009 International Conference on Intelligent Engineering Systems.

[31]  Daniel A. Menascé,et al.  A method for evaluating the impact of software configuration parameters on e-commerce sites , 2005, WOSP '05.

[32]  Marin Litoiu,et al.  A performance analysis method for autonomic computing systems , 2007, TAAS.

[33]  Daniel A. Menascé,et al.  On the Use of Performance Models to Design Self-Managing Computer Systems , 2003, Int. CMG Conference.

[34]  Jerome A. Rolia,et al.  The Method of Layers , 1995, IEEE Trans. Software Eng..

[35]  Asser N. Tantawi,et al.  Performance management for cluster-based web services , 2005, IEEE Journal on Selected Areas in Communications.