An Empirical Study of Bugs in Software Build Systems

Build system converts source code, libraries and other data into executable programs by orchestrating the execution of compilers and other tools. The whole building process is managed by a software build system, such as Make, Ant, CMake, Maven, Scons, and QMake. The reliability of software build systems would affect the reliability of the build process. In this paper, we perform an empirical study on bugs in software build systems. We analyze four software build systems, Ant, Maven, CMake and QMake, which are four typical and widely-used software build systems, and can be used to build Java, C, C++ systems. We investigate their bug database and code repositories, randomly sample a set of bug reports and their fixes (800 bugs reports totally, and 199, 250, 200, and 151 bug reports for Ant, Maven, CMake and QMake, respectively), and manually assign them into various categories. We find that 21.35% of the bugs belong to the external interface category, 18.23% of the bugs belong to the logic category, and 12.86% of the bugs belong to the configuration category. We also investigate the relationship between bug categories and bug severities.

[1]  Ying Zou,et al.  An empirical study of build system migrations in practice: Case studies on KDE and the Linux kernel , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[2]  Daniel M. Germán,et al.  Towards a simplification of the bug report form in eclipse , 2008, MSR '08.

[3]  Shane McIntosh,et al.  An empirical study of build maintenance effort , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[4]  Wolfgang De Meuter,et al.  Design recovery and maintenance of build systems , 2007, 2007 IEEE International Conference on Software Maintenance.

[5]  Seth Hallem,et al.  An Empiri al Study of Operating Systems , 2001 .

[6]  Michael W. Godfrey,et al.  The build-time software architecture view , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[7]  Forrest Shull,et al.  Defect categorization: making use of a decade of widely varying historical data , 2008, ESEM '08.

[8]  Peter Smith.  Software Build Systems: Principles and Experience , 2011 .

[9]  Michael W. Godfrey,et al.  Build system issues in multilanguage software , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[10]  David Lo,et al.  An Empirical Study of Bugs in Machine Learning Systems , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[11]  Junfeng Yang,et al.  An empirical study of operating systems errors , 2001, SOSP.

[12]  Wolfgang De Meuter,et al.  The Evolution of the Linux Build System , 2007, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..

[13]  T Epperly,et al.  Software in the DOE: The Hidden Overhead of''The Build'' , 2002 .

[14]  Ahmed E. Hassan,et al.  Security versus performance bugs: a case study on Firefox , 2011, MSR '11.

[15]  Sunghun Kim,et al.  Toward an understanding of bug fix patterns , 2009, Empirical Software Engineering.