On Rapid Releases and Software Testing

Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases. We analyze the results of 312,502 execution runs of the 1,547 mostly manual system level test cases of Mozilla Fire fox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulated our findings with a Mozilla QA engineer. In rapid releases, testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases.

[1]  Shorten releaSe cycleS by bringing developerS to application lifecycle management , 2011 .

[2]  Douglas A. Wolfe,et al.  Nonparametric Statistical Methods , 1973 .

[3]  Claes Wohlin,et al.  The effect of moving from a plan-driven to an incremental software development approach with agile practices , 2010, Empirical Software Engineering.

[4]  Claes Wohlin,et al.  A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case , 2009, J. Syst. Softw..

[5]  Tore Dybå,et al.  Transition from a plan-driven process to Scrum: a longitudinal case study on software quality , 2010, ESEM '10.

[6]  C. Borror.  Nonparametric Statistical Methods, 2nd, Ed. , 2001 .

[7]  Alan W. Brown.  A Case Study in Agile-at-Scale Delivery , 2011, XP.

[8]  Foutse Khomh,et al.  Do faster releases improve software quality? An empirical case study of Mozilla Firefox , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[9]  Aniruddha S. Gokhale,et al.  Techniques and processes for improving the quality and performance of open-source software , 2006, Softw. Process. Improv. Pract..

[10]  Kent Beck,et al.  Extreme Programming Explained: Embrace Change (2nd Edition) , 2004 .

[11]  Michael W. Godfrey,et al.  A tale of two browsers , 2011, MSR '11.

[12]  Kent L. Beck,et al.  Extreme programming explained - embrace change , 1990 .

[13]  Ossi Taipale,et al.  A study on agility and testing processes in software organizations , 2010, ISSTA '10.

[14]  Erich Gamma.  Agile, open source, distributed, and on-time: inside the eclipse development process , 2005, ICSE '05.

[15]  Jez Humble,et al.  Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation , 2010 .