How has SPI changed in times of agile development? Results from a multi‐method study

The emergence of agile methods and practices has not only changed the development processes but might also have affected how companies conduct software process improvement (SPI). Through a set of complementary studies, we aim to understand how SPI has changed in times of agile software development. Specifically, we aim (a) to identify and characterize the set of publications that connect elements of agility to SPI, (b) to explore to which extent agile methods/practices have been used in the context of SPI, and (c) to understand whether the topics addressed in the literature are relevant and useful for industry professionals. To study these questions, we conducted an in‐depth analysis of the literature identified in a previous mapping study, an interview study, and an analysis of the responses given by industry professionals to SPI‐related questions stemming from an independently conducted survey study. Regarding the first question, we identified 55 publications that focus on both SPI and agility of which 48 present and discuss how agile methods/practices are used to steer SPI initiatives. Regarding the second question, we found that the two most frequently mentioned agile methods in the context of SPI are Scrum and Extreme Programming (XP), while the most frequently mentioned agile practices are integrate often, test‐first, daily meeting, pair programming, retrospective, on‐site customer, and product backlog. Regarding the third question, we found that a majority of the interviewed and surveyed industry professionals see SPI as a continuous activity. They agree with the agile SPI literature that agile methods/practices play an important role in SPI activities but that the importance given to specific agile methods/practices does not always coincide with the frequency with which these methods/practices are mentioned in the literature.

[1]  Rory O'Connor,et al.  Assessing the Value of an Agile Coach in Agile Method Adoption , 2014, EuroSPI.

[2]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[3]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[4]  Marco Kuhrmann,et al.  Systematic Software Development: A State of the Practice Report from Germany , 2015, 2015 IEEE 10th International Conference on Global Software Engineering.

[5]  Jürgen Münch,et al.  Software Process Definition and Management , 2012, The Fraunhofer IESE Series on Software and Systems Engineering.

[6]  Marjan Hericko,et al.  PROCESSUS – Integration of SEI CMM and ISO quality models , 2004, Software Quality Journal.

[7]  Victor R. Basili,et al.  Learning Through Application: The Maturing of the QIP in the SEL , 2011, Making Software.

[8]  Eric S. K. Yu,et al.  Capitalizing on Empirical Evidence during Agile Adoption , 2010, 2010 Agile Conference.

[9]  Jürgen Münch,et al.  Experiences and results from tailoring and deploying a large process standard in a company , 2008, Softw. Process. Improv. Pract..

[10]  Watts S. Humphrey Personal Software Process (PSP) , 2000 .

[11]  Thomas Birkhölzer,et al.  A Framework for Systematic Evaluation of Process Improvement Priorities , 2011, 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications.

[12]  Hans van Vliet,et al.  When agile meets the enterprise , 2013, Inf. Softw. Technol..

[13]  Onur Demirörs,et al.  Assessment of Agile Maturity Models: A Multiple Case Study , 2013, SPICE.

[14]  Lars Mathiassen,et al.  Making SPI Happen: The Roads to Process Implementation , 2004, ECIS.

[15]  D. Ross Jeffery,et al.  An exploratory study of why organizations do not adopt CMMI , 2007, J. Syst. Softw..

[16]  Tracy Hall,et al.  De-motivators for software process improvement: an analysis of practitioners' views , 2003, J. Syst. Softw..

[17]  Javier Garzás,et al.  A case study of software process improvement with CMMI‐DEV and Scrum in Spanish companies , 2013, J. Softw. Evol. Process..

[18]  J. G. Brodman,et al.  What small businesses and small organizations say about the CMM , 1994, Proceedings of 16th International Conference on Software Engineering.

[19]  Rory O'Connor,et al.  Investigating software process in practice: A grounded theory perspective , 2008, J. Syst. Softw..

[20]  Dietmar Pfahl,et al.  What Do We Know about Scientific Software Development's Agile Practices? , 2012, Computing in Science & Engineering.

[21]  Mohd Syazwan Abdullah,et al.  A proposed methodology for establishing software process development improvement for small software development firms , 2011, WCIT.

[22]  Risto Nevalainen,et al.  Agile maturity model: analysing agile maturity characteristics from the SPICE perspective , 2014, J. Softw. Evol. Process..

[23]  Abdul Azim Abdul Ghani,et al.  Exploring Key Factors of Pilot Projects in Agile Transformation Process Using a Grounded Theory Study , 2013, ICIST.

[24]  Zhang Lina,et al.  Research on Combining Scrum with CMMI in Small and Medium Organizations , 2012, 2012 International Conference on Computer Science and Electronics Engineering.

[25]  Ivan A. Garcia,et al.  Using a web-based tool to define and implement software process improvement initiatives in a small industrial setting , 2010, IET Softw..

[26]  Davi Viana,et al.  The influence of human aspects on software process improvement: Qualitative research findings and comparison to previous studies , 2012, EASE.

[27]  Stefan Wagner,et al.  What Do Practitioners Vary in Using Scrum? , 2015, XP.

[28]  Michael Felderer,et al.  On the Role of Software Quality Management in Software Process Improvement , 2016, PROFES.

[29]  M. Elizabeth C. Hull,et al.  Software development processes - an assessment , 2002, Inf. Softw. Technol..

[30]  Marco Kuhrmann,et al.  Software process improvement: where is the evidence?: initial findings from a systematic mapping study , 2015, ICSSP.

[31]  Tore Dybå,et al.  Transition from a plan-driven process to Scrum: a longitudinal case study on software quality , 2010, ESEM '10.

[32]  Mira Kajko-Mattsson,et al.  Guiding the adoption of software development methods , 2014, ICSSP 2014.

[33]  Watts S. Humphrey,et al.  Team Software Process (TSP) , 2000 .

[34]  Marco Kuhrmann,et al.  Experiences from the Design of an Artifact Model for Distributed Agile Project Management , 2014, 2014 IEEE 9th International Conference on Global Software Engineering.

[35]  Charles W. Butler,et al.  Choice of Software Development Methodologies: Do Organizational, Project, and Team Characteristics Matter? , 2016, IEEE Software.

[36]  Maya Daneva,et al.  On the pragmatic design of literature studies in software engineering: an experience-based guideline , 2016, Empirical Software Engineering.

[37]  Scott Ambler,et al.  Agility at scale: Economic governance, measured improvement, and disciplined delivery , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[38]  F. Song,et al.  Dissemination and publication of research findings: an updated review of related biases. , 2010, Health technology assessment.

[39]  Claes Wohlin,et al.  Software process improvement through the Lean Measurement (SPI-LEAM) method , 2010, J. Syst. Softw..

[40]  Ivan Rozman,et al.  Managing the complexity of SPI in small companies , 2000, Softw. Process. Improv. Pract..

[41]  Fergal McCaffery,et al.  Adopting agile practices when developing software for use in the medical domain , 2014, J. Softw. Evol. Process..

[42]  Hazura Zulzalil,et al.  Agile transformation: What is it about? , 2014, 2014 8th. Malaysian Software Engineering Conference (MySEC).

[43]  Roel Wieringa,et al.  Requirements engineering paper classification and evaluation criteria: a proposal and a discussion , 2005, Requirements Engineering.

[44]  Anita D. Carleton,et al.  Case studies of software-process-improvement measurement , 1994, Computer.

[45]  Amr Noaman Abdel-Hamid,et al.  Process Increments: An Agile Approach to Software Process Improvement , 2011, 2011 AGILE Conference.

[46]  Kirsi Korhonen Evaluating the impact of an agile transformation: a longitudinal case study in a distributed context , 2012, Software Quality Journal.

[47]  Silvio Romero de Lemos Meira,et al.  Using CMMI together with agile software development: A systematic review , 2015, Inf. Softw. Technol..

[48]  Silvio Romero de Lemos Meira,et al.  A Reference Model for Agile Quality Assurance: Combining Agile Methodologies and Maturity Models , 2014, 2014 9th International Conference on the Quality of Information and Communications Technology.

[49]  Geir Kjetil Hanssen,et al.  Process fusion: An industrial case study on agile software product line engineering , 2008, J. Syst. Softw..

[50]  Pekka Abrahamsson,et al.  An iterative improvement process for agile software development , 2007, Softw. Process. Improv. Pract..

[51]  Andreia Malucelli,et al.  Processes versus people: How should agile software development maturity be defined? , 2014, J. Syst. Softw..

[52]  Kurt Schneider,et al.  Breaking the ice for agile development of embedded software: an industry experience report , 2004, Proceedings. 26th International Conference on Software Engineering.

[53]  James D. Arthur,et al.  A disciplined approach to adopting agile practices: the agile adoption framework , 2007, Innovations in Systems and Software Engineering.

[54]  Marco Kuhrmann,et al.  Software process improvement: a systematic mapping study on the state of the art , 2016, PeerJ Comput. Sci..

[55]  Wil M. P. van der Aalst,et al.  Agile development with software process mining , 2014, ICSSP 2014.

[56]  Burak Turhan,et al.  Impact of process conformance on the effects of test-driven development , 2014, ESEM '14.

[57]  Tomas Schweigert,et al.  TestSPICE and Agile Testing - Synergy or Confusion , 2013, SPICE.

[58]  Victor R. Basili,et al.  The Experience Factory and Its Relationship to Other Quality Approaches , 1995, Adv. Comput..

[59]  Mary Shaw,et al.  Writing good software engineering research papers: minitutorial , 2003, ICSE 2003.

[60]  Pekka Abrahamsson,et al.  Agile methods rapidly replacing traditional methods at Nokia: A survey of opinions on agile transformation , 2011, Inf. Softw. Technol..

[61]  Mary Shaw,et al.  Writing good software engineering research papers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[62]  Tracy Hall,et al.  Motivators of Software Process Improvement: an analysis of practitioners' views , 2002, Journal of Systems and Software.

[63]  Jarmo J. Ahonen,et al.  LAPPI: A light‐weight technique to practical process modeling and improvement target identification , 2013, J. Softw. Evol. Process..

[64]  Austen Rainer,et al.  Implementing software process improvement: an empirical study , 2002, Softw. Process. Improv. Pract..

[65]  Daniela Cruzes,et al.  Interpretative case studies on agile team productivity and management , 2013, Inf. Softw. Technol..

[66]  Mario Piattini,et al.  Using Scrum to guide the execution of software process improvement in small organizations , 2010, J. Syst. Softw..

[67]  Khaled El Emam,et al.  An empirical review of software process assessments , 2000, Adv. Comput..

[68]  Alfonso Fuggetta,et al.  Software process: a roadmap , 2000, ICSE '00.

[69]  Mikio Aoyama,et al.  Beyond software factories: concurrent-development process and an evolution of software process technology in Japan , 1996, Inf. Softw. Technol..

[70]  Hans Christian Benestad,et al.  Perceived productivity threats in large agile development projects , 2010, ESEM '10.

[71]  Hwan-Seung Yong,et al.  Agile Software Development Framework in a Small Project Environment , 2013, J. Inf. Process. Syst..

[72]  Werner Mellis,et al.  Success factors of organizational change in software process improvement , 1998, Softw. Process. Improv. Pract..

[73]  Marco Kuhrmann,et al.  Teaching Empirical Software Engineering Using Expert Teams , 2017, SEUH.

[74]  Tony Gorschek,et al.  A method for evaluating rigor and industrial relevance of technology evaluations , 2011, Empirical Software Engineering.

[75]  Ian Allison Organizational Factors Shaping Software Process Improvement in Small-Medium Sized Software Teams: A Multi-Case Analysis , 2010, 2010 Seventh International Conference on the Quality of Information and Communications Technology.

[76]  Pekka Abrahamsson,et al.  Integrating agile software development and software process improvement: a longitudinal case study , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[77]  Claude Y. Laporte,et al.  Systems and Software Engineering Standards for Very Small Entities: Implementation and Initial Results , 2014, 2014 9th International Conference on the Quality of Information and Communications Technology.

[78]  Huynh Quyet Thang,et al.  Combining maturity with agility: lessons learnt from a case study , 2013, SoICT '13.

[79]  Naji Habra,et al.  AM-QuICk: A Measurement-Based Framework for Agile Methods Customisation , 2013, 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement.

[80]  Risto Nevalainen,et al.  Agile Maturity Model: A Synopsis as a First Step to Synthesis , 2013, EuroSPI.

[81]  Horst Lichter,et al.  SPIALS: A light-weight Software Process Improvement self-assessment tool , 2011, 2011 Malaysian Conference in Software Engineering.

[82]  Mario Piattini,et al.  Software process improvement in small and medium software enterprises: a systematic review , 2008, Software Quality Journal.

[83]  Marco Kuhrmann,et al.  Is Water-Scrum-Fall Reality? On the Use of Agile and Traditional Development Practices , 2015, PROFES.

[84]  Rory V. O'Connor,et al.  Software Development Processes for Games: A Systematic Literature Review , 2014, EuroSPI.

[85]  Paolo Tell,et al.  How Does Software Process Improvement Address Global Software Engineering? , 2016, 2016 IEEE 11th International Conference on Global Software Engineering (ICGSE).

[86]  Marco Kuhrmann,et al.  SPI is Dead, isn't it? Clear the Stage for Continuous Learning! , 2019, 2019 IEEE/ACM International Conference on Software and System Processes (ICSSP).

[87]  Philippe Kruchten,et al.  Contextualizing agile software development , 2013, J. Softw. Evol. Process..

[88]  Stefan,et al.  The European IS Profession in the Global Networking Environment , 2004 .

[89]  João Araújo,et al.  Generating feature model from creative requirements using model driven design , 2012, SPLC '12.

[90]  Outi Salo,et al.  Improving software process in agile software development projects: results from two XP case studies , 2004, Proceedings. 30th Euromicro Conference, 2004..

[91]  Fergal McCaffery,et al.  Balancing Agility and Discipline in a Medical Device Software Organisation , 2013, SPICE.

[92]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[93]  Brian Henderson-Sellers,et al.  Method Engineering: Theory and Practice , 2006, ISTA.

[94]  Reidar Conradi,et al.  Improving Software Process Improvement , 2002, IEEE Softw..

[95]  Andreas S. Andreou,et al.  Evidence of Agile Adoption in Software Organizations: An Empirical Survey , 2013, EuroSPI.

[96]  Fergal McCaffery,et al.  Ahaa --agile, hybrid assessment method for automotive, safety critical smes , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[97]  Mario Piattini,et al.  A Process for Driving Process Improvement in VSEs , 2009, ICSP.

[98]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[99]  Jil Klünder,et al.  Walking Through the Method Zoo: Does Higher Education Really Meet Software Industry Demands? , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET).

[100]  Brian Henderson-Sellers,et al.  A framework to support the evaluation, adoption and improvement of agile methods in practice , 2008, J. Syst. Softw..

[101]  Andreia Malucelli,et al.  Progressive Outcomes: A framework for maturing in agile software development , 2015, J. Syst. Softw..

[102]  Torgeir Dingsøyr,et al.  A decade of agile methodologies: Towards explaining agile software development , 2012, J. Syst. Softw..

[103]  Pekka Abrahamsson,et al.  Strengths and barriers behind the successful agile deployment—insights from the three software intensive companies in Finland , 2011, Empirical Software Engineering.

[104]  David Lorge Parnas,et al.  A rational design process: How and why to fake it , 1986, IEEE Transactions on Software Engineering.

[105]  Tony Gorschek,et al.  A practitioner's guide to light weight software process assessment and improvement planning , 2008, J. Syst. Softw..

[106]  Des Greer,et al.  Preparing small software companies for tailored agile method adoption: Minimally intrusive risk assessment , 2008, Softw. Process. Improv. Pract..

[107]  Fabio Kon,et al.  Experiences tracking agile projects: an empirical study , 2006, Journal of the Brazilian Computer Society.

[108]  María José Escalona Cuaresma,et al.  A scrum-based approach to CMMI maturity level 2 in web development environments , 2012, IIWAS '12.

[109]  Pasi Kuvaja,et al.  Combining Lean Thinking and Agile Methods for Software Development: A Case Study of a Finnish Provider of Wireless Embedded Systems Detailed , 2014, 2014 47th Hawaii International Conference on System Sciences.

[110]  Carolyn B. Seaman,et al.  Why do programmers avoid metrics? , 2008, ESEM '08.

[111]  Jürgen Münch,et al.  How has SPI changed in times of agile development? Results from a multi-method study , 2020, SE.

[112]  Paul Hodgetts Refactoring the development process: experiences with the incremental adoption of agile practices , 2004, Agile Development Conference.

[113]  Vahid Garousi,et al.  Hybrid Software Development Approaches in Practice: A European Perspective , 2019, IEEE Software.

[114]  Watts S. Humphrey,et al.  Managing the software process , 1989, The SEI series in software engineering.

[115]  Martin Höst,et al.  A review of methods for evaluation of maturity models for process improvement , 2012, J. Softw. Evol. Process..

[116]  Vahid Garousi,et al.  Hybrid software and system development in practice: waterfall, scrum, and beyond , 2017, ICSSP.