Usage, costs, and benefits of continuous integration in open-source projects

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? Without answering such questions, developers, tool builders, and researchers make decisions based on folklore instead of data. In this paper, we use three complementary methods to study the usage of CI in open-source projects. To understand which CI systems developers use, we analyzed 34,544 open-source projects from GitHub. To understand how developers use CI, we analyzed 1,529,291 builds from the most commonly used CI system. To understand why projects use or do not use CI, we surveyed 442 developers. With this data, we answered several key questions related to the usage, costs, and benefits of CI. Among our results, we show evidence that supports the claim that CI helps projects release more often, that CI is widely adopted by the most popular projects, as well as finding that the overall percentage of projects using CI continues to grow, making it important and timely to focus more research on CI.

[1]  Sebastian Erdweg,et al.  A sound and optimal incremental build system with dynamic dependencies , 2015, OOPSLA 2015.

[2]  Mika Mäntylä,et al.  The highways and country roads to continuous deployment , 2015, IEEE Software.

[3]  Nico L. U. van Meeteren, Paul J. M. Helders.  Why? , 2000 .

[4]  Myra B. Cohen,et al.  Continuous test suite augmentation in software product lines , 2013, SPLC '13.

[5]  Beryl Plimmer,et al.  Ambient awareness of build status in collocated software teams , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[6]  Klaas-Jan Stol,et al.  Scaling agile methods to regulated environments: An industry case study , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[7]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[8]  Miroslaw Staron,et al.  Supporting Continuous Integration by Code-Churn Based Test Selection , 2015, 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering.

[9]  Georgios Gousios,et al.  How (Much) Do Developers Test? , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[10]  Stefan Biffl,et al.  Communicating continuous integration servers for increasing effectiveness of automated testing , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[11]  Maria Paasivaara,et al.  Stakeholder Perceptions of the Adoption of Continuous Integration -- A Case Study , 2015, 2015 Agile Conference.

[12]  Harald C. Gall,et al.  SQA-Profiles: Rule-based activity profiles for Continuous Integration environments , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[13]  Grady Booch,et al.  Object-Oriented Design with Applications , 1990 .

[14]  David Lo,et al.  An Empirical Study on the Adequacy of Testing in Open Source Projects , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[15]  Harald C. Gall,et al.  Supporting continuous integration by mashing-up software quality information , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[16]  Hung Viet Nguyen,et al.  Detecting semantic changes in Makefile build code , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[17]  Ahmet Çelik,et al.  Build system with lazy retrieval for Java projects , 2016, SIGSOFT FSE.

[18]  Kent L. Beck,et al.  Embracing Change with Extreme Programming , 1999, Computer.

[19]  Darko Marinov,et al.  Practical regression test selection with dynamic file dependencies , 2015, ISSTA.

[20]  Georgios Gousios,et al.  The GHTorent dataset and tool suite , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[21]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[22]  Yuriy Brun,et al.  Data debugging with continuous testing , 2013, ESEC/FSE 2013.

[23]  Jafar M. Al-Kofahi,et al.  Extracting Configuration Knowledge from Build Files with Symbolic Analysis , 2015, 2015 IEEE/ACM 3rd International Workshop on Release Engineering.

[24]  Jakob Engblom.  Virtual to the (near) end - Using virtual platforms for continuous integration , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[25]  Michael D. Ernst,et al.  Continuous testing in Eclipse , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[26]  Georgios Gousios,et al.  Oops, my tests broke the build: An analysis of Travis CI builds with GitHub , 2016, PeerJ Prepr..

[27]  Premkumar T. Devanbu,et al.  Quality and productivity outcomes relating to continuous integration in GitHub , 2015, ESEC/SIGSOFT FSE.

[28]  Ade Miller,et al.  A Hundred Days of Continuous Integration , 2008, Agile 2008 Conference.

[29]  Rui Abreu,et al.  Continuous test generation: enhancing continuous integration with automated test generation , 2014, ASE.

[30]  Carlos Jensen,et al.  Gender Differences in Early Free and Open Source Software Joining Process , 2012, OSS.