Understanding the impact of rapid releases on software quality

Many software companies are shifting from the traditional multi-month release cycle to shorter release cycles. For example, Google Chrome and Mozilla Firefox release new versions every 6 weeks. These shorter release cycles reduce the users’ waiting time for a new release and offer better feedback and marketing opportunities to companies, but it is unclear if the quality of the software product improves as well, since developers and testers are under more pressure. In this paper, we extend our previous empirical study of Mozilla Firefox on the impact of rapid releases on quality assurance with feedback by Mozilla project members. The study compares crash rates, median uptime, and the proportion of pre- and post-release bugs in traditional releases with those in rapid releases, and we also analyze the source code changes made by developers to identify potential changes in the development process. We found that (1) with shorter release cycles, users do not experience significantly more pre- or post-release bugs (percentage-wise) and (2) bugs are fixed faster, yet (3) users experience these bugs earlier during software execution (the program crashes earlier). Increased integration activity and propagation of harder bugs to later versions account for some of these findings. Overall, our case study suggests that a clear release engineering process with thorough automation is one of the major challenges when switching to rapid releases.

[1]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[2]  Prakash Ramaswamy,et al.  The effects of individual XP practices on software development effort , 2003, SOEN.

[3]  J. W. Johnson.  A Heuristic Method for Estimating the Relative Weight of Predictor Variables in Multiple Regression , 2000, Multivariate behavioral research.

[4]  Erich Gamma.  Agile, open source, distributed, and on-time: inside the eclipse development process , 2005, ICSE '05.

[5]  Vikram S. Adve,et al.  An empirical study of reported bugs in server software with implications for automated bug diagnosis , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[6]  Alan W. Brown.  A Case Study in Agile-at-Scale Delivery , 2011, XP.

[7]  Damon Afkari,et al.  ? ? ? ? ? ? ? ? ? ? ? ? ? 30 ? ? ? ? ? ? ? ? ? ? ? ? ? ? , 2011 .

[8]  Foutse Khomh,et al.  Do faster releases improve software quality? An empirical case study of Mozilla Firefox , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[9]  Katherine J. Stewart,et al.  Observations on Patterns of Development in Open Source Software Projects , 2005 .

[10]  Shorten releaSe cycleS by bringing developerS to application lifecycle management , 2011 .

[11]  Katherine J. Stewart,et al.  Observations on patterns of development in open source software projects , 2005, ACM SIGSOFT Softw. Eng. Notes.

[12]  Andreas Zeller,et al.  When do changes induce fixes? , 2005, ACM SIGSOFT Softw. Eng. Notes.

[13]  Slinger Jansen,et al.  Ten Misconceptions about Product Software Release Management explained using Update Cost/Value Functions , 2006, 2006 International Workshop on Software Product Management (IWSPM'06 - RE'06 Workshop).

[14]  Jez Humble,et al.  Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation , 2010 .

[15]  J. Fox,et al.  Applied Regression Analysis and Generalized Linear Models , 2008 .

[16]  Andrew Glover,et al.  Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series) , 2007 .

[17]  Julie E. Kendall,et al.  The Challenge of Improving Software Quality: Developers' Beliefs about the Contribution of Agile Practices , 2009, AMCIS.

[18]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[19]  C. Borror.  Nonparametric Statistical Methods, 2nd, Ed. , 2001 .

[20]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[21]  Reidar Conradi.  7th international workshop on software configuration management (SCM7), Boston, 18–19 May 1997 , 1997, ICSE '97.

[22]  Ahmed E. Hassan,et al.  A qualitative study on performance bugs , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[23]  Foutse Khomh,et al.  On Rapid Releases and Software Testing , 2013, 2013 IEEE International Conference on Software Maintenance.

[24]  Aniruddha S. Gokhale,et al.  Techniques and processes for improving the quality and performance of open-source software , 2006, Softw. Process. Improv. Pract..

[25]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[26]  Merijn de Jonge,et al.  Nix: A Safe and Policy-Free System for Software Deployment , 2004, LISA.

[27]  Matthias Marschall,et al.  Transforming a Six Month Release Cycle to Continuous Flow , 2007, Agile 2007 (AGILE 2007).

[28]  Yuanyuan Zhou,et al.  Learning from mistakes: a comprehensive study on real world concurrency bug characteristics , 2008, ASPLOS.

[29]  Ahmed E. Hassan,et al.  Impact of Installation Counts on Perceived Quality: A Case Study on Debian , 2011, 2011 18th Working Conference on Reverse Engineering.

[30]  Brendan Murphy,et al.  Characterizing the differences between pre- and post- release versions of software , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[31]  Michael W. Godfrey,et al.  A tale of two browsers , 2011, MSR '11.

[32]  Tijs van der Storm,et al.  Continuous release and upgrade of component-based software , 2005, SCM '05.

[33]  Denise Phillips,et al.  Extreme Adoption Experiences of a B2B Start-up , 2002 .

[34]  D. Wolfe,et al.  Nonparametric Statistical Methods. , 1974 .

[35]  Heinz D. Knoell,et al.  Applied Quality Assurance Methods under the Open Source Development Model , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[36]  David Lo,et al.  An Empirical Study of Bugs in Machine Learning Systems , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[37]  Foutse Khomh,et al.  An Entropy Evaluation Approach for Triaging Field Crashes: A Case Study of Mozilla Firefox , 2011, 2011 18th Working Conference on Reverse Engineering.