Releasing fast and slow: an exploratory case study at ING

The appeal of delivering new features faster has led many software projects to adopt rapid releases. However, it is not well understood what the effects of this practice are. This paper presents an exploratory case study of rapid releases at ING, a large banking company that develops software solutions in-house, to characterize rapid releases. Since 2011, ING has shifted to a rapid release model. This switch has resulted in a mixed environment of 611 teams releasing relatively fast and slow. We followed a mixed-methods approach in which we conducted a survey with 461 participants and corroborated their perceptions with 2 years of code quality data and 1 year of release delay data. Our research shows that: rapid releases are more commonly delayed than their non-rapid counterparts, however, rapid releases have shorter delays; rapid releases can be beneficial in terms of reviewing and user-perceived quality; rapidly released software tends to have a higher code churn, a higher test coverage and a lower average complexity; challenges in rapid releases are related to managing dependencies and certain code aspects, e.g., design debt.

[1]  Mika Mäntylä,et al.  Prioritizing Manual Test Cases in Traditional and Rapid Release Environments , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[2]  Andy Zaidman,et al.  Continuous Delivery Practices in a Large Financial Organization , 2016, ICSME.

[3]  A. Strauss,et al.  Grounded theory , 2017 .

[4]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[5]  Andrew Begel,et al.  Usage and Perceptions of Agile Software Development in an Industrial Context: An Exploratory Study , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[6]  Uirá Kulesza,et al.  The Impact of Switching to a Rapid Release Cycle on the Integration Delay of Addressed Issues - An Empirical Study of the Mozilla Firefox Project , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[7]  Martin C. Rinard,et al.  The Challenges of Staying Together While Moving Fast: An Exploratory Study , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[8]  Ian Alexander,et al.  An introduction to qualitative research , 2000, Eur. J. Inf. Syst..

[9]  Brian Fitzgerald,et al.  Why and How Should Open Source Projects Adopt Time-Based Releases? , 2015, IEEE Software.

[10]  Robert L. Nord,et al.  A study of enabling factors for rapid fielding combined practices to balance speed and stability , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[11]  Foutse Khomh,et al.  On rapid releases and software testing: a case study and a semi-systematic literature review , 2015, Empirical Software Engineering.

[12]  Richard Torkar,et al.  Adopting Free/Libre/Open Source Software Practices, Techniques and Methods for Industrial Use , 2011, J. Assoc. Inf. Syst..

[13]  Zadia Codabux,et al.  Managing technical debt: An industrial case study , 2013, 2013 4th International Workshop on Managing Technical Debt (MTD).

[14]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[15]  Rashmi Popli,et al.  Cost and effort estimation in agile software development , 2014, 2014 International Conference on Reliability Optimization and Information Technology (ICROIT).

[16]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[17]  Uirá Kulesza,et al.  An Empirical Study of Delays in the Integration of Addressed Issues , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[18]  Des Greer,et al.  Quantitative studies in software release planning under risk and resource constraints , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[19]  Foutse Khomh,et al.  Factors impacting rapid releases: an industrial case study , 2014, ESEM '14.

[20]  Shane McIntosh,et al.  Modern Release Engineering in a Nutshell -- Why Researchers Should Care , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[21]  Chandrasekar Subramaniam,et al.  Determinants of open source software project success: A longitudinal study , 2009, Decis. Support Syst..

[22]  A. Furnham.  Response bias, social desirability and dissimulation , 1986 .

[23]  Arie van Deursen,et al.  Software Ecosystem Call Graph for Dependency Management , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[24]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[25]  Robert J. Eisenberg.  A threshold based approach to technical debt , 2012, SOEN.

[26]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[27]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[28]  Matt Blaze,et al.  Moving Targets: Security and Rapid-Release in Firefox , 2014, CCS.

[29]  Tore Dybå,et al.  Transition from a plan-driven process to Scrum: a longitudinal case study on software quality , 2010, ESEM '10.

[30]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[31]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[32]  Foutse Khomh,et al.  Understanding the impact of rapid releases on software quality , 2015, Empirical Software Engineering.

[33]  Tom Mens,et al.  An empirical comparison of dependency network evolution in seven software packaging ecosystems , 2017, Empirical Software Engineering.

[34]  Peng Liang,et al.  A systematic mapping study on technical debt and its management , 2015, J. Syst. Softw..

[35]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[36]  Foutse Khomh,et al.  Do faster releases improve software quality? An empirical case study of Mozilla Firefox , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[37]  Claes Wohlin,et al.  The effect of moving from a plan-driven to an incremental software development approach with agile practices , 2010, Empirical Software Engineering.

[38]  B SeamanCarolyn.  Qualitative Methods in Empirical Studies of Software Engineering , 1999 .

[39]  Aniruddha S. Gokhale,et al.  Techniques and processes for improving the quality and performance of open-source software , 2006, Softw. Process. Improv. Pract..

[40]  Matthias Marschall,et al.  Transforming a Six Month Release Cycle to Continuous Flow , 2007, Agile 2007 (AGILE 2007).

[41]  Michel R. V. Chaudron,et al.  A Survey of the Practice of Design -- Code Correspondence amongst Professional Software Engineers , 2007, ESEM 2007.

[42]  Tom Mens,et al.  An empirical comparison of dependency issues in OSS packaging ecosystems , 2017, 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER).