Quality-Aware DevOps Research: Where Do We Stand?

DevOps is an emerging paradigm that reduces the barriers between developers and operations teams to offer continuous fast delivery and enable quick responses to changing requirements within the software life cycle. A significant volume of activity has been carried out in recent years with the aim of coupling DevOps stages with tools and methods to improve the quality of the produced software and the underpinning delivery methodology. While the research community has produced a sustained effort by conducting numerous studies and innovative development tools to support quality analyses within DevOps, there is still a limited cohesion between the research themes in this domain and a shortage of surveys that holistically examine quality engineering work within DevOps. In this paper, we address the gap by comprehensively surveying existing efforts in this area, categorizing them according to the stage of the DevOps lifecycle to which they primarily contribute. The survey holistically spans across all the DevOps stages, identify research efforts to improve architectural design, modeling and infrastructure-as-code, continuous-integration/continuous-delivery (CI/CD), testing and verification, and runtime management. Our analysis also outlines possible directions for future work in quality-aware DevOps, looking in particular at AI for DevOps and DevOps for AI software.

[1]  Damian Andrew Tamburri,et al.  Towards Omnia: A Monitoring Factory for Quality-Aware DevOps , 2017, ICPE Companion.

[2]  Chris Parnin,et al.  The Seven Sins: Security Smells in Infrastructure as Code Scripts , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[3]  Marin Litoiu,et al.  A DevOps framework for quality-driven self-protection in web software systems , 2018, CASCON.

[4]  Anne Koziolek,et al.  Continuous Integration of Performance Model , 2018, ICPE Companion.

[5]  Paul Dourish,et al.  "Breaking the code", moving between private and public work in collaborative software development , 2003, GROUP.

[6]  Stefano Russo,et al.  Towards Continuous Software Reliability Testing in DevOps , 2019, 2019 IEEE/ACM 14th International Workshop on Automation of Software Test (AST).

[7]  José Merseguer,et al.  Quality Assessment in DevOps: Automated Analysis of a Tax Fraud Detection System , 2017, ICPE Companion.

[8]  Holger Karl,et al.  Profile your chains, not functions: Automated network service profiling in DevOps environments , 2017, 2017 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN).

[9]  Prem Prakash Jayaraman,et al.  A Framework for Monitoring Microservice-Oriented Cloud Applications in Heterogeneous Virtualization Environments , 2019, 2019 IEEE 12th International Conference on Cloud Computing (CLOUD).

[10]  Hans-Arno Jacobsen,et al.  HyScale: Hybrid and Network Scaling of Dockerized Microservices in Cloud Data Centres , 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[11]  Rami Bahsoon,et al.  Software Architecture for Big Data and the Cloud , 2017 .

[12]  Fabienne Boyer,et al.  Architecture-Based Automated Updates of Distributed Microservices , 2018, ICSOC.

[13]  Manish Virmani,et al.  Understanding DevOps & bridging the gap from continuous integration to continuous delivery , 2015, Fifth International Conference on the Innovative Computing Technology (INTECH 2015).

[14]  Chetan Bansal,et al.  Building Sankie: An AI Platform for DevOps , 2019, 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE).

[15]  Robert Heinrich,et al.  An Architectural Model-Based Approach toQuality-aware DevOps in Cloud Applications , 2017 .

[16]  Erkuden Rios,et al.  Continuous Deployment of Trustworthy Smart IoT Systems , 2020, J. Object Technol..

[17]  Peter W. O'Hearn,et al.  Continuous Reasoning: Scaling the impact of formal methods , 2018, LICS.

[18]  Alexander Romanovsky,et al.  Fault tolerant internet computing: Benchmarking and modelling trade-offs between availability, latency and consistency , 2019, J. Netw. Comput. Appl..

[19]  Heiko Koziolek,et al.  A Model Transformation from the Palladio Component Model to Layered Queueing Networks , 2008, SIPEW.

[20]  André van Hoorn,et al.  Exploiting DevOps Practices for Dependable and Secure Continuous Delivery Pipelines , 2018, 2018 IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE).

[21]  Marin Litoiu,et al.  Delivering Elastic Containerized Cloud Applications to Enable DevOps , 2017, 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[22]  Laurie A. Williams,et al.  Where Are The Gaps? A Systematic Mapping Study of Infrastructure as Code Research , 2018, Inf. Softw. Technol..

[23]  Liming Zhu,et al.  The Intersection of Continuous Deployment and Architecting Process: Practitioners' Perspectives , 2016, ESEM.

[24]  Rick Kazman,et al.  Architectural Support for DevOps in a Neo-Metropolis BDaaS Platform , 2015, 2015 IEEE 34th Symposium on Reliable Distributed Systems Workshop (SRDSW).

[25]  Michael Le,et al.  Container and Microservice Driven Design for Cloud Infrastructure DevOps , 2016, 2016 IEEE International Conference on Cloud Engineering (IC2E).

[26]  James R. Larus,et al.  Righting software , 2004, IEEE Software.

[27]  Ciera Jaspan,et al.  Lessons from building static analysis tools at Google , 2018, Commun. ACM.

[28]  Cesare Pautasso,et al.  Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences , 2019, ICPE.

[29]  Tonghoon Suk,et al.  Failure-Aware Application Placement Modeling and Optimization in High Turnover DevOps Environment , 2019, 2019 IEEE 12th International Conference on Cloud Computing (CLOUD).

[30]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[31]  Tobias Schneider.  Achieving Cloud Scalability with Microservices and DevOps in the Connected Car Domain , 2016, Software Engineering.

[32]  Lianping Chen,et al.  The Journal of Systems and Software , 2022 .

[33]  Samuel Kounev,et al.  Chamulteon: Coordinated Auto-Scaling of Micro-Services , 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[34]  Elisabetta Di Nitto,et al.  A software architecture framework for quality-aware DevOps , 2016, QUDOS@ISSTA.

[35]  Li-Te Cheng,et al.  Sometimes you need to see through walls: a field study of application programming interfaces , 2004, CSCW.

[36]  Roy Meissner,et al.  Using DevOps Principles to Continuously Monitor RDF Data Quality , 2016, SEMANTICS.

[37]  Cole Schlesinger,et al.  One-Click Formal Methods , 2019, IEEE Software.

[38]  Ioana-Raluca Adochiei,et al.  Innovative Devops for Artificial Intelligence , 2019, The Scientific Bulletin of Electrical Engineering Faculty.

[39]  Peng Huang,et al.  AIOps: Real-World Challenges and Research Innovations , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).

[40]  Nelly Bencomo,et al.  An Architectural Framework for Quality-Driven Adaptive Continuous Experimentation , 2019, 2019 IEEE/ACM Joint 4th International Workshop on Rapid Continuous Software Engineering and 1st International Workshop on Data-Driven Decisions, Experimentation and Evolution (RCoSE/DDrEE).

[41]  Frank Leymann,et al.  DevOpSlang - Bridging the Gap between Development and Operations , 2014, ESOCC.

[42]  André van Hoorn,et al.  Scalability Assessment of Microservice Architecture Deployment Configurations: A Domain-based Approach Leveraging Operational Profiles and Load Tests , 2020, J. Syst. Softw..

[43]  Wilhelm Hasselbring,et al.  Microservice Architectures for Scalability, Agility and Reliability in E-Commerce , 2017, 2017 IEEE International Conference on Software Architecture Workshops (ICSAW).

[44]  Danilo Ardagna,et al.  A Model-Driven DevOps Framework for QoS-Aware Cloud Applications , 2015, 2015 17th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC).

[45]  Valeria Cardellini,et al.  Horizontal and Vertical Scaling of Container-Based Applications Using Reinforcement Learning , 2019, 2019 IEEE 12th International Conference on Cloud Computing (CLOUD).

[46]  Valeria Cardellini,et al.  Hierarchical Scaling of Microservices in Kubernetes , 2020, 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS).

[47]  Harald C. Gall,et al.  We're doing it live: A multi-method empirical study on continuous experimentation , 2018, Inf. Softw. Technol..

[48]  Harald C. Gall,et al.  Bifrost: Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies , 2016, Middleware.

[49]  André van Hoorn,et al.  Microservice-Tailored Generation of Session-Based Workload Models for Representative Load Testing , 2019, 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS).

[50]  Valeria Cardellini,et al.  Geo-distributed efficient deployment of containers with Kubernetes , 2020, Comput. Commun..

[51]  André van Hoorn,et al.  Towards Automating Representative Load Testing in Continuous Software Engineering , 2018, ICPE Companion.

[52]  Rick Kazman,et al.  Agile Big Data Analytics Development: An Architecture-Centric Approach , 2016, 2016 49th Hawaii International Conference on System Sciences (HICSS).

[53]  Yuming Zhou,et al.  The impact of continuous integration on other software development practices: A large-scale empirical study , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[54]  Yuanyuan Zhang,et al.  Multi-objective Optimisation of Online Distributed Software Update for DevOps in Clouds , 2019, ACM Trans. Internet Techn..

[55]  Damian A. Tamburri,et al.  TOSCA-based Intent modelling: goal-modelling for infrastructure-as-code , 2019, SICS Software-Intensive Cyber-Physical Systems.

[56]  Liming Zhu,et al.  Non-Intrusive Anomaly Detection With Streaming Performance Metrics and Logs for DevOps in Public Clouds: A Case Study in AWS , 2016, IEEE Transactions on Emerging Topics in Computing.

[57]  Wilhelm Hasselbring,et al.  Performance-oriented DevOps: A Research Agenda , 2015, ArXiv.

[58]  Geoffrey C. Fox,et al.  HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack , 2015, 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[59]  Alim Ul Gias,et al.  ATOM: Model-Driven Autoscaling for Microservices , 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[60]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[61]  Peter W. O'Hearn,et al.  From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis , 2018, 2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[62]  Pooyan Jamshidi,et al.  Microservices Architecture Enables DevOps: Migration to a Cloud-Native Architecture , 2016, IEEE Software.

[63]  José Carlos Bregieiro Ribeiro,et al.  Improving La Redoute's CI/CD Pipeline and DevOps Processes by Applying Machine Learning Techniques , 2018, 2018 11th International Conference on the Quality of Information and Communications Technology (QUATIC).

[64]  Carlo Ghezzi,et al.  Supporting Self-Adaptation via Quantitative Verification and Sensitivity Analysis at Run Time , 2016, IEEE Transactions on Software Engineering.

[65]  Liming Zhu,et al.  DevOps - A Software Architect's Perspective , 2015, SEI series in software engineering.

[66]  Cesare Pautasso,et al.  Towards Holistic Continuous Software Performance Assessment , 2017, ICPE Companion.

[67]  Cristina L. Abad,et al.  A Case for Performance-Aware Deployment of Containers , 2019, WOC@Middleware.

[68]  Subho Sankar Banerjee,et al.  FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices , 2020, OSDI.