Statistical Inference of Software Performance Models for Parametric Performance Completions

Software performance engineering (SPE) enables software architects to ensure high performance standards for their applications. However, applying SPE in practice is still challenging. Most enterprise applications include a large software basis, such as middleware and legacy systems. In many cases, the software basis is the determining factor of the system’s overall timing behavior, throughput, and resource utilization. To capture these influences on the overall system’s performance, established performance prediction methods (model-based and analytical) rely on models that describe the performance-relevant aspects of the system under study. Creating such models requires detailed knowledge on the system’s structure and behavior that, in most cases, is not available. In this paper, we abstract from the internal structure of the system under study. We focus on message-oriented middleware (MOM) and analyze the dependency between the MOM’s usage and its performance. We use statistical inference to conclude these dependencies from observations. For ActiveMQ 5.3, the resulting functions predict the performance with a relative mean square error 0.1.

[1]  Samuel Kounev,et al.  Article in Press Performance Evaluation Performance Evaluation of Message-oriented Middleware Using the Specjms2007 Benchmark , 2022 .

[2]  C. Murray Woodside,et al.  Using regression splines for software performance analysis , 2000, WOSP '00.

[3]  Jens Happe,et al.  State dependence in performance evaluation of component-based software systems , 2010, WOSP/SIPEW '10.

[4]  Steffen Becker,et al.  Parametric performance completions for model-driven performance prediction , 2010, Perform. Evaluation.

[5]  Heiko Koziolek,et al.  Performance evaluation of component-based software systems: A survey , 2010, Perform. Evaluation.

[6]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[7]  Stephen Dawson,et al.  Estimating service resource consumption from response time measurements , 2009, VALUETOOLS.

[8]  Eugene Miya,et al.  On "Software engineering" , 1985, SOEN.

[9]  Samuel Kounev,et al.  Benchmarking Publish/Subscribe-Based Messaging Systems , 2010, DASFAA Workshops.

[10]  David Beasley,et al.  An overview of genetic algorithms: Part 1 , 1993 .

[11]  Steffen Becker,et al.  The Palladio component model for model-driven performance prediction , 2009, J. Syst. Softw..

[12]  Ramesh Govindan,et al.  Automatic request categorization in internet services , 2008, PERV.

[13]  Elaine J. Weyuker,et al.  Software performance testing based on workload characterization , 2002, WOSP '02.

[14]  Jerome A. Rolia,et al.  Characterizing the scalability of a large web-based shopping system , 2001, ACM Trans. Internet Techn..

[15]  Giovanni Denaro,et al.  Early performance testing of distributed software applications , 2004, WOSP '04.

[16]  Lucia Happe,et al.  Automated Feature Model-Based Generation of Refinement Transformations , 2009, 2009 35th Euromicro Conference on Software Engineering and Advanced Applications.

[17]  Hui Li,et al.  Black-box performance models: prediction based on observation , 2009, QUASOSS '09.

[18]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[19]  Jerome H. Friedman,et al.  Rejoinder: Multivariate Adaptive Regression Splines , 1991 .

[20]  Asser N. Tantawi,et al.  Dynamic estimation of CPU demand of web traffic , 2006, valuetools '06.

[21]  Alan Fekete,et al.  Design-level performance prediction of component-based applications , 2005, IEEE Transactions on Software Engineering.

[22]  Paola Inverardi,et al.  Model-based performance prediction in software development: a survey , 2004, IEEE Transactions on Software Engineering.

[23]  Connie U. Smith,et al.  Performance Engineering of Software Systems , 1990, SIGMETRICS Perform. Evaluation Rev..

[24]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[25]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[26]  Scott Barber.  Creating effective load models for performance testing with incomplete empirical data , 2004, Proceedings. Sixth IEEE International Workshop on Web Site Evolution.

[27]  Asser N. Tantawi,et al.  Enhanced inferencing: estimation of a workload dependent performance model , 2009, VALUETOOLS.

[28]  Samuel Kounev,et al.  Benchmarking and Performance Modeling of Event-Based SystemsModellierung und Bewertung von Ereignis-basierten Systemen , 2009, it Inf. Technol..

[29]  Marin Litoiu,et al.  Performance Model Estimation and Tracking Using Optimal Filters , 2008, IEEE Transactions on Software Engineering.

[30]  Samuel Kounev,et al.  Automated extraction of palladio component models from running enterprise Java applications , 2009, VALUETOOLS.

[31]  J. Galletly.  An Overview of Genetic Algorithms , 1992 .

[32]  D. B. Davis,et al.  Sun Microsystems Inc. , 1993 .

[33]  John Hunt,et al.  Java Message Service (JMS) , 2003 .

[34]  Ralf H. Reussner,et al.  Using Genetic Search for Reverse Engineering of Parametric Behavior Models for Performance Prediction , 2010, IEEE Transactions on Software Engineering.