A sound and optimal incremental build system with dynamic dependencies

Build systems are used in all but the smallest software projects to invoke the right build tools on the right files in the right order. A build system must be sound (after a build, generated files consistently reflect the latest source files) and efficient (recheck and rebuild as few build units as possible). Contemporary build systems provide limited efficiency because they lack support for expressing fine-grained file dependencies. We present a build system called pluto that supports the definition of reusable, parameterized, interconnected builders. When run, a builder notifies the build system about dynamically required and produced files as well as about other builders whose results are needed. To support fine-grained file dependencies, we generalize the traditional notion of time stamps to allow builders to declare their actual requirements on a file's content. pluto collects the requirements and products of a builder with their stamps in a build summary. This enables pluto to provides provably sound and optimal incremental rebuilding. To support dynamic dependencies, our rebuild algorithm interleaves dependency analysis and builder execution and enforces invariants on the dependency graph through a dynamic analysis. We have developed pluto as a Java API and used it to implement more than 25 builders. We describe our experience with migrating a larger Ant build script to pluto and compare the respective build times.

[1]  Eelco Visser,et al.  Building program optimizers with rewriting strategies , 1998, ICFP '98.

[2]  Allan Heydon,et al.  Caching function calls using precise dependencies , 2000, PLDI '00.

[3]  Allan Heydon,et al.  The Vesta Approach to Software Configuration Management , 2001 .

[4]  T Epperly,et al.  Software in the DOE: The Hidden Overhead of''The Build'' , 2002 .

[5]  Davide Ancona,et al.  Polymorphic bytecode: compositional compilation for Java-like languages , 2005, POPL '05.

[6]  Eelco Visser,et al.  The spoofax language workbench: rules for declarative specification of languages and IDEs , 2010, OOPSLA.

[7]  Sebastian Erdweg,et al.  Featherweight TeX and Parser Correctness , 2010, SLE.

[8]  Shane McIntosh,et al.  The evolution of ANT build systems , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[9]  Neil Mitchell Shake before building: replacing make with haskell , 2012, ICFP '12.

[10]  Gail C. Murphy,et al.  An exploratory study of forces and frictions affecting large-scale model-driven development , 2012, MODELS'12.

[11]  Sebastian Erdweg,et al.  A framework for extensible languages , 2013, GPCE '13.

[12]  Sebastian Erdweg,et al.  Extensible languages for flexible and principled domain abstraction , 2013 .

[13]  K. Rustan M. Leino,et al.  Formalizing and Verifying a Modern Build Language , 2014, FM.

[14]  Benjamin Livshits,et al.  Automated migration of build scripts using dynamic analysis and search-based refactoring , 2014, OOPSLA.