Performance-oriented DevOps: A Research Agenda

DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area.

[1]  Susan L. Graham,et al.  Gprof: A call graph execution profiler , 1982, SIGPLAN '82.

[2]  Ray Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[3]  Paul C. Jorgensen,et al.  Object-oriented integration testing , 1994, CACM.

[4]  Jerome A. Rolia,et al.  Parameter estimation for performance models of distributed application systems , 1995, CASCON.

[5]  Virgílio A. F. Almeida,et al.  A methodology for workload characterization of E-commerce sites , 1999, EC '99.

[6]  Jerome A. Rolia,et al.  Trace-Based Load Characterization for Gernerating Performance Software Models , 1999, IEEE Trans. Software Eng..

[7]  C. Murray Woodside,et al.  Using regression splines for software performance analysis , 2000, WOSP '00.

[8]  Daniel A. Menascé,et al.  A Method for Design and Performance Modeling of Client/Server Systems , 2000, IEEE Trans. Software Eng..

[9]  Connie U. Smith,et al.  Software performance antipatterns , 2000, WOSP '00.

[10]  David J. Lilja,et al.  Measuring computer performance : A practitioner's guide , 2000 .

[11]  Raffaela Mirandola,et al.  Deriving a queueing network based performance model from UML diagrams , 2000, WOSP '00.

[12]  C. H. Ward,et al.  Section I References , 2000 .

[13]  Gregor Kiczales,et al.  Aspect-oriented programming , 2001, ESEC/FSE-9.

[14]  Connie U. Smith,et al.  Software Performance AntiPatterns; Common Performance Problems and their Solutions , 2001, Int. CMG Conference.

[15]  Virgílio A. F. Almeida,et al.  Capacity Planning for Web Services: Metrics, Models, and Methods , 2001 .

[16]  Meir M. Lehman,et al.  Rules and Tools for Software Evolution Planning and Management , 2001, Ann. Softw. Eng..

[17]  Nenad Medvidovic,et al.  Focus: a light-weight, incremental approach to software architecture recovery and evolution , 2001, Proceedings Working IEEE/IFIP Conference on Software Architecture.

[18]  Lennard Kerber.  Scenario-Based Performance Evaluation of SDL/MSC-Specified Systems , 2001, Performance Engineering.

[19]  Daniel A. Menascé,et al.  TPC-W: A Benchmark for E-Commerce , 2002, IEEE Internet Comput..

[20]  Ali Shokoufandeh,et al.  The Sisyphus database retrieval software performance antipattern , 2002, WOSP '02.

[21]  Dorina C. Petriu,et al.  Software Performance Models from System Scenarios in Use Case Maps , 2002, Computer Performance Evaluation / TOOLS.

[22]  Mark S. Squillante,et al.  Workload service requirements analysis: a queueing network optimization approach , 2002, Proceedings. 10th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunications Systems.

[23]  Connie U. Smith,et al.  New Software Performance AntiPatterns: More Ways to Shoot Yourself in the Foot , 2002, Int. CMG Conference.

[24]  Zhao Li,et al.  Testing the suitability of Markov chains as Web usage models , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[25]  Connie U. Smith,et al.  More New Software Antipatterns: Even More Ways to Shoot Yourself in the Foot , 2003, Int. CMG Conference.

[26]  Dorina C. Petriu,et al.  Performance Analysis with UML , 2003, UML for Real.

[27]  Te-Kai Liu,et al.  A capacity sizing tool for a business process integration middleware , 2004, Proceedings. IEEE International Conference on e-Commerce Technology, 2004. CEC 2004..

[28]  Paola Inverardi,et al.  Model-based performance prediction in software development: a survey , 2004, IEEE Transactions on Software Engineering.

[29]  Adrian Mos.  A Framework for Adaptive Monitoring and Performance Management of Component-Based Enterprise Applications , 2004 .

[30]  Daniel A. Menascé,et al.  Composing Web Services: A QoS View , 2004, IEEE Internet Comput..

[31]  T. Parsons.  A Framework for Automatically Detecting and Assessing Performance Antipatterns in Component Based Systems using Run-Time Analysis , 2004 .

[32]  C. Murray Woodside,et al.  Performance modeling from software components , 2004, WOSP '04.

[33]  Brian King.  Performance Assurance for IT Systems , 2004 .

[34]  Richard Mortier,et al.  Using Magpie for Request Extraction and Workload Modelling , 2004, OSDI.

[35]  Scott Barber.  Creating effective load models for performance testing with incomplete empirical data , 2004, Proceedings. Sixth IEEE International Workshop on Web Site Evolution.

[36]  Manish Gupta,et al.  Problem Determination Using Dependency Graphs and Run-Time Behavior Models , 2004, DSOM.

[37]  Fan Zhang,et al.  Parameter inference of queueing models for IT systems using end-to-end measurements , 2004, SIGMETRICS '04/Performance '04.

[38]  José Merseguer,et al.  Performance by unified model analysis (PUMA) , 2005, WOSP '05.

[39]  Daniel A. Menascé,et al.  On the Use of Online Analytic Performance Models, in Self-Managing and Self-Organizing Computer Systems , 2005, Self-star Properties in Complex Information Systems.

[40]  A. Petrenko,et al.  Dynamic analysis of java applications for multithreaded antipatterns , 2005, WODA '05.

[41]  Petr Tuma,et al.  Repeated results analysis for middleware regression benchmarking , 2005, Perform. Evaluation.

[42]  Armando Fox,et al.  Detecting application-level failures in component-based Internet services , 2005, IEEE Transactions on Neural Networks.

[43]  Diwakar Krishnamurthy,et al.  A model-based approach for testing the performance of web applications , 2006, SOQUA '06.

[44]  Jerome A. Rolia,et al.  A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems , 2006, IEEE Transactions on Software Engineering.

[45]  Adam Wierman,et al.  Open Versus Closed: A Cautionary Tale , 2006, NSDI.

[46]  Lionel C. Briand,et al.  Toward the Reverse Engineering of UML Sequence Diagrams for Distributed Java Software , 2006, IEEE Transactions on Software Engineering.

[47]  Asser N. Tantawi,et al.  Analytic modeling of multitier Internet applications , 2007, TWEB.

[48]  C. Murray Woodside,et al.  Interaction tree algorithms to extract effective architecture and layered performance models from traces , 2007, J. Syst. Softw..

[49]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[50]  Vittorio Cortellessa,et al.  A Framework for Automated Generation of Architectural Feedback from Software Performance Analysis , 2007, EPEW.

[51]  Paolo Cremonesi,et al.  How to Select Significant Workloads in Performance Models , 2007, Int. CMG Conference.

[52]  Liming Zhu,et al.  Revel8or: Model Driven Capacity Planning Tool Suite , 2007, 29th International Conference on Software Engineering (ICSE'07).

[53]  Daniel A. Menascé.  Computing Missing Service Demand Parameters for Performance Models , 2008, Int. CMG Conference.

[54]  Asser N. Tantawi,et al.  CPU demand for web serving: Measurement analysis and dynamic estimation , 2008, Perform. Evaluation.

[55]  Mourad Oussalah,et al.  Extraction of Component-Based Architecture from Object-Oriented Systems , 2007, Seventh Working IEEE/IFIP Conference on Software Architecture (WICSA 2008).

[56]  Evgenia Smirni,et al.  Analysis of application performance and its change via representative application signatures , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[57]  Marin Litoiu,et al.  Performance Model Estimation and Tracking Using Optimal Filters , 2008, IEEE Transactions on Software Engineering.

[58]  Wilhelm Hasselbring,et al.  Generating Probabilistic and Intensity-Varying Workload for Web-Based Software Systems , 2008, SIPEW.

[59]  Jing Xu,et al.  Rule-based automatic software performance diagnosis and improvement , 2008, WOSP '08.

[60]  Ramesh Govindan,et al.  Automatic request categorization in internet services , 2008, PERV.

[61]  Evgenia Smirni,et al.  Anomaly? application change? or workload change? towards automated detection of application performance anomaly and change , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[62]  Roberto Turrin,et al.  Robust Workload Estimation in Queueing Network Performance Models , 2008, 16th Euromicro Conference on Parallel, Distributed and Network-Based Processing (PDP 2008).

[63]  Raghunath Othayoth Nambiar,et al.  Energy cost, the key challenge of today's data centers: a power consumption analysis of TPC-C results , 2008, Proc. VLDB Endow..

[64]  Ralf Reussner,et al.  ByCounter: Portable Runtime Counting of Bytecode Instructions and Method Invocations , 2008 .

[65]  Wilhelm Hasselbring,et al.  Automatic Failure Diagnosis Support in Distributed Large-Scale Software Systems Based on Timing Behavior Anomaly Correlation , 2009, 2009 13th European Conference on Software Maintenance and Reengineering.

[66]  Samuel Kounev,et al.  Automated extraction of palladio component models from running enterprise Java applications , 2009, VALUETOOLS.

[67]  Asser N. Tantawi,et al.  Real-time performance modeling for adaptive software systems , 2009, VALUETOOLS.

[68]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[69]  Evgenia Smirni,et al.  Automated anomaly detection and performance modeling of enterprise applications , 2009, TOCS.

[70]  Steffen Becker,et al.  The Palladio component model for model-driven performance prediction , 2009, J. Syst. Softw..

[71]  Stephen Dawson,et al.  Estimating service resource consumption from response time measurements , 2009, VALUETOOLS.

[72]  Asser N. Tantawi,et al.  Enhanced inferencing: estimation of a workload dependent performance model , 2009, VALUETOOLS.

[73]  Ladan Tahvildari,et al.  Self-adaptive software: Landscape and research challenges , 2009, TAAS.

[74]  LIKWID: A Lightweight Performance-Oriented Tool Suite for x86 Multicore Environments , 2010, 2010 39th International Conference on Parallel Processing Workshops.

[75]  Heiko Koziolek,et al.  Performance evaluation of component-based software systems: A survey , 2010, Perform. Evaluation.

[76]  Paolo Cremonesi,et al.  Service Time Estimation with a Refinement Enhanced Hybrid Clustering Algorithm , 2010, ASMTA.

[77]  Miroslaw Malek,et al.  A survey of online failure prediction methods , 2010, CSUR.

[78]  Will Cappelli.  Magic Quadrant for Application Performance Monitoring , 2010 .

[79]  Peter Nobel,et al.  Practical performance models for complex, popular applications , 2010, SIGMETRICS '10.

[80]  Sam Houston.  Department of Computer Science , 2010 .

[81]  Vittorio Cortellessa,et al.  A Process to Effectively Identify "Guilty" Performance Antipatterns , 2010, FASE.

[82]  Hui Li,et al.  SLA-driven planning and optimization of enterprise applications , 2010, WOSP/SIPEW '10.

[83]  Abdelhak-Djamel Seriai,et al.  Component-Based Architecture Recovery from Object Oriented Systems via Relational Concept Analysis , 2010, CLA.

[84]  Steffen Becker,et al.  Reverse Engineering Component Models for Quality Predictions , 2010, 2010 14th European Conference on Software Maintenance and Reengineering.

[85]  Klaus Krogmann,et al.  Reconstruction of software component architectures and behaviour models using static and dynamic analysis , 2012 .

[86]  Michael I. Jordan,et al.  Bayesian inference for queueing networks and modeling of internet services , 2010, 1001.3355.

[87]  Thomas Vogel,et al.  The Role of Models and Megamodels at Runtime , 2010, MoDELS.

[88]  Ramón Puigjaner,et al.  Performance Model Interchange Format (PMIF 2): A comprehensive approach to Queueing Network Model interoperability , 2010, Perform. Evaluation.

[89]  Jerome A. Rolia,et al.  MODE: Mix Driven On-line Resource Demand Estimation , 2011, 2011 7th International Conference on Network and Service Management.

[90]  Samuel Kounev,et al.  Automated simulation-based capacity planning for enterprise data fabrics , 2011, SimuTools.

[91]  Anne Koziolek,et al.  Automated Improvement of Software Architecture Models for Performance and Other Quality Attributes , 2011 .

[92]  Gregory R. Ganger,et al.  Diagnosing Performance Changes by Comparing Request Flows , 2011, NSDI.

[93]  Wilhelm Hasselbring,et al.  Self-adaptive software system monitoring for performance anomaly localization , 2011, ICAC '11.

[94]  Samuel Kounev,et al.  Automated extraction of architecture-level performance models of distributed component-based systems , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[95]  Anne Koziolek,et al.  Detection and solution of software performance antipatterns in palladio architectural models , 2011, ICPE '11.

[96]  Markus von Detten.  Archimetrix: A Tool for Deficiency-Aware Software Architecture Reconstruction , 2012, WCRE.

[97]  Petr Tuma,et al.  Capturing performance assumptions using stochastic performance logic , 2012, ICPE '12.

[98]  Wilhelm Hasselbring,et al.  Kieker: a framework for application performance monitoring and dynamic software analysis , 2012, ICPE '12.

[99]  Jun Wei,et al.  Application-Level CPU Consumption Estimation: Towards Performance Isolation of Multi-tenancy Web Applications , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[100]  Rouven Krebs,et al.  Automated inference of goal-oriented performance prediction functions , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[101]  Stefan Becker,et al.  Workload-aware System Monitoring Using Performance Predictions Applied to a Large-scale E-Mail System , 2012, 2012 Joint Working IEEE/IFIP Conference on Software Architecture and European Conference on Software Architecture.

[102]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[103]  Chen Fu,et al.  Automatically finding performance problems with feedback-directed learning software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[104]  J. LaFountain.  Inc. , 2013, American Art.

[105]  Gene Kim,et al.  The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win , 2013 .

[106]  Vítor Estêvão Silva Souza,et al.  Engineering adaptation with Zanshin: An experience report , 2013, 2013 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[107]  Giuliano Casale,et al.  An Offline Demand Estimation Method for Multi-threaded Applications , 2013, 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems.

[108]  Martin Moser,et al.  Systematic performance evaluation based on tailored benchmark applications , 2013, ICPE '13.

[109]  Giuliano Casale,et al.  Bayesian Service Demand Estimation Using Gibbs Sampling , 2013, 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems.

[110]  Sebastian Lehrig,et al.  CloudScale: scalability management for cloud systems , 2013, ICPE '13.

[111]  Helmut Krcmar,et al.  Automatic Performance Model Generation for Java Enterprise Edition (EE) Applications , 2013, EPEW.

[112]  Dennis Westermann,et al.  Deriving Goal-oriented Performance Models by Systematic Experimentation , 2013 .

[113]  Jens Happe,et al.  Supporting swift reaction: Automatically uncovering performance problems by systematic experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[114]  Lars Grunske,et al.  Software Architecture Optimization Methods: A Systematic Literature Review , 2013, IEEE Transactions on Software Engineering.

[115]  Helmut Krcmar,et al.  Integrating the Palladio-Bench into the Software Development Process of a SOA Project , 2013, KPDAYS.

[116]  Fredrik Abbors,et al.  Model-based performance testing in the cloud using the mbpet tool , 2013, ICPE '13.

[117]  Samuel Kounev,et al.  LibReDE: a library for resource demand estimation , 2014, ICPE.

[118]  Thomas Vogel,et al.  Model-Driven Engineering of Self-Adaptive Software with EUREMA , 2014, ACM Trans. Auton. Adapt. Syst..

[119]  Danilo Ardagna,et al.  Quality-of-service in cloud computing: modeling techniques and their applications , 2014, Journal of Internet Services and Applications.

[120]  Roozbeh Farahbod,et al.  Automatic detection of performance anti-patterns in inter-component communications , 2014, QoSA '14.

[121]  Fabian Brosig,et al.  Architecture-Level Software Performance Models for Online Performance Prediction , 2014 .

[122]  Helmut Krcmar,et al.  Using architecture-level performance models as resource profiles for enterprise applications , 2014, QoSA '14.

[123]  Petr Tuma.  Performance awareness: keynote abstract , 2014, ICPE.

[124]  Samuel Kounev,et al.  Modeling variations in load intensity over time , 2014, LT '14.

[125]  Helmut Krcmar,et al.  Towards Performance Awareness in Java EE Development Environments , 2014, SoSP.

[126]  Wilhelm Hasselbring,et al.  Integrating Run-time Observations and Design Component Models for Cloud System Analysis , 2014, Models@run.time.

[127]  Petr Tuma,et al.  Towards Performance-Aware Engineering of Autonomic Component Ensembles , 2014, ISoLA.

[128]  Helmut Krcmar,et al.  Performance Management Work , 2014, Bus. Inf. Syst. Eng..

[129]  Samuel Kounev,et al.  Evaluating approaches to resource demand estimation , 2015, Perform. Evaluation.

[130]  Antonín Steinhauser,et al.  DOs and DON'Ts of Conducting Performance Measurements in Java , 2015, ICPE.

[131]  André van Hoorn,et al.  Model-driven online capacity management for component-based software systems , 2014, Softwaretechnik-Trends.

[132]  Giuliano Casale,et al.  Estimating Computational Requirements in Multi-Threaded Applications , 2015, IEEE Transactions on Software Engineering.

[133]  Andreas Hotho,et al.  Modeling and Extracting Load Intensity Profiles , 2017, 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems.

[134]  Wilhelm Hasselbring,et al.  Automatic Extraction of Probabilistic Workload Specifications for Load Testing Session-Based Application Systems , 2015, EAI Endorsed Trans. Self Adapt. Syst..

[135]  Wilhelm Hasselbring,et al.  Including Performance Benchmarks into Continuous Integration to Enable DevOps , 2015, SOEN.

[136]  Antonín Steinhauser,et al.  Utilizing Performance Unit Tests To Increase Performance Awareness , 2015, ICPE.

[137]  L. Carvajal,et al.  IEEE Transactions on Software Engineering , 2016 .

[138]  Uwe Fink,et al.  Performance Solutions A Practical Guide To Creating Responsive Scalable Software , 2016 .

[139]  Hans-Peter Fröschle.  DevOps , 2017, HMD Praxis der Wirtschaftsinformatik.

[140]  A. Shamsai,et al.  Multi-objective Optimization , 2017, Encyclopedia of Machine Learning and Data Mining.