The impact of continuous integration on other software development practices: A large-scale empirical study

Continuous Integration (CI) has become a disruptive innovation in software development: with proper tool support and adoption, positive effects have been demonstrated for pull request throughput and scaling up of project sizes. As any other innovation, adopting CI implies adapting existing practices in order to take full advantage of its potential, and "best practices" to that end have been proposed. Here we study the adaptation and evolution of code writing and submission, issue and pull request closing, and testing practices as Travis CI is adopted by hundreds of established projects on GitHub. To help essentialize the quantitative results, we also survey a sample of GITHUB developers about their experiences with adopting Travis CI. Our findings suggest a more nuanced picture of how GitHub teams are adapting to, and benefiting from, continuous integration technology than suggested by prior work.

[1]  Kelly Blincoe,et al.  The Sky Is Not the Limit: Multitasking Across GitHub Projects , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[2]  Jan Bosch,et al.  Automated software integration flows in industry: a multiple-case study , 2014, ICSE Companion.

[3]  Daniela E. Damian,et al.  The promises and perils of mining GitHub , 2009, MSR 2014.

[4]  Harald C. Gall,et al.  Comparing fine-grained source code changes and code churn for bug prediction , 2011, MSR '11.

[5]  Danny Dig,et al.  How do centralized and distributed version control systems impact software changes? , 2014, ICSE.

[6]  Gang Yin,et al.  Determinants of pull-based development in the context of continuous integration , 2016, Science China Information Sciences.

[7]  Mike Loukides,et al.  What is DevOps , 2012 .

[8]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[9]  Jesper Holck,et al.  Continuous Integration and Quality Assurance: a case study of two open source projects , 2003, Australas. J. Inf. Syst..

[10]  Leif Singer,et al.  Creating a shared understanding of testing culture on a social coding site , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[11]  Philipp Leitner,et al.  An Empirical Analysis of Build Failures in the Continuous Integration Workflows of Java-Based Open-Source Software , 2017, 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR).

[12]  Georgios Gousios,et al.  Oops, my tests broke the build: An analysis of Travis CI builds with GitHub , 2016, PeerJ Prepr..

[13]  Maria Paasivaara,et al.  Stakeholder Perceptions of the Adoption of Continuous Integration -- A Case Study , 2015, 2015 Agile Conference.

[14]  Arie van Deursen,et al.  An exploratory study of the pull-based software development model , 2014, ICSE.

[15]  Abigail Morrison,et al.  Increasing quality and managing complexity in neuroinformatics software development with continuous integration , 2013, Front. Neuroinform..

[16]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[17]  Yiqun Li,et al.  Implementation and Evaluation of Automatic Prioritization for Continuous Integration Test Cases , 2016 .

[18]  Walter Brenner,et al.  Fit for Continuous Integration: How Organizations Assimilate an Agile Practice , 2014, AMCIS.

[19]  Roel Wieringa,et al.  Abductive Inference Design , 2014 .

[20]  Georgios Gousios,et al.  TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration , 2017, 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR).

[21]  Premkumar T. Devanbu,et al.  Wait for It: Determinants of Pull Request Evaluation Latency on GitHub , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[22]  Mika Mäntylä,et al.  The highways and country roads to continuous deployment , 2015, IEEE Software.

[23]  Shinichi Nakagawa,et al.  A general and simple method for obtaining R2 from generalized linear mixed‐effects models , 2013 .

[24]  Sean Stolberg,et al.  Enabling Agile Testing through Continuous Integration , 2009, 2009 Agile Conference.

[25]  Eray Tüzün,et al.  Using continuous integration and automated test techniques for a robust C4ISR system , 2009, 2009 24th International Symposium on Computer and Information Sciences.

[26]  Alexander Serebrenik,et al.  Continuous Integration in a Social-Coding World: Empirical Evidence from GitHub , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[27]  Alexander Serebrenik,et al.  A Data Set for Social Diversity Studies of GitHub Teams , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[28]  Rui Abreu,et al.  Continuous test generation: enhancing continuous integration with automated test generation , 2014, ASE.

[29]  Premkumar T. Devanbu,et al.  Quality and productivity outcomes relating to continuous integration in GitHub , 2015, ESEC/SIGSOFT FSE.

[30]  Ade Miller,et al.  A Hundred Days of Continuous Integration , 2008, Agile 2008 Conference.

[31]  Jan Bosch,et al.  Continuous Integration Applied to Software-Intensive Embedded Systems - Problems and Experiences , 2016, PROFES.

[32]  Andrew Glover,et al.  Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series) , 2007 .

[33]  Mathias Meyer,et al.  Continuous Integration and Its Tools , 2014, IEEE Software.

[34]  Alastair Smith,et al.  So You Say You Want a Revolution , 1999 .

[35]  Michael Gertz,et al.  Mining email social networks , 2006, MSR '06.

[36]  James Roche,et al.  Adopting DevOps practices in quality assurance , 2013, CACM.

[37]  Georgios Gousios,et al.  GHTorrent: Github's data from a firehose , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[38]  Jan Bosch,et al.  EXPERIENCED BENEFITS OF CONTINUOUS INTEGRATION IN INDUSTRY SOFTWARE PRODUCT DEVELOPMENT: A CASE STUDY , 2013, ICSE 2013.

[39]  Dirk Riehle,et al.  Continuous Integration in Open Source Software Development , 2008, OSS.

[40]  Darko Marinov,et al.  Continuous Integration (CI) Needs and Wishes for Developers of Proprietary Code , 2016 .

[41]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[42]  Teng Long,et al.  Collaborative testing across shared software components (doctoral symposium) , 2015, ISSTA.

[43]  Fabrizio Cannizzo,et al.  Pushing the Boundaries of Testing and Continuous Integration , 2008, Agile 2008 Conference.

[44]  Paul C. Johnson.  Extension of Nakagawa & Schielzeth's R2GLMM to random slopes models , 2014, Methods in ecology and evolution.

[45]  Darko Marinov,et al.  Usage, costs, and benefits of continuous integration in open-source projects , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[46]  A K Wagner,et al.  Segmented regression analysis of interrupted time series studies in medication use research , 2002, Journal of clinical pharmacy and therapeutics.

[47]  Richard Berntsson-Svensson,et al.  Challenges When Adopting Continuous Integration: A Case Study , 2014, PROFES.

[48]  Jan Bosch,et al.  Modeling continuous integration practice differences in industry software development , 2014, J. Syst. Softw..

[49]  Georgios Gousios,et al.  Work practices and challenges in pull-based development: the contributor's perspective , 2015, ICSE.

[50]  Christian Berger,et al.  Visualizing Testing Activities to Support Continuous Integration: A Multiple Case Study , 2014, XP.

[51]  Carl Adams.  Managing crowdsourcing assignments , 2011 .

[52]  Shawn A. Bohner,et al.  The impact of agile methods on software project management , 2005, 12th IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'05).

[53]  Stefan Biffl,et al.  Communicating continuous integration servers for increasing effectiveness of automated testing , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.