Whispec: white-box testing of libraries using declarative specifications

We present a novel framework, Whispec, for white-box testing of methods that manipulate structurally complex data, such as those that pervade library classes. Given method preconditions as declarative constraints, our framework systematically generates test inputs for the methods to maximize their code coverage. The constraints are written in Alloy, a first-order language based on relations. To test a method, given its precondition constraint, we first solve that constraint using the Alloy Analyzer and translate a solution into a test input. Next, we execute the method on that input and build the path condition for the resulting execution path. Then, we run the analyzer on a conjunction of the precondition and a new path condition that represents a previously unexplored path. The solution is translated to a new test input, which triggers the next round of test generation. The iterative execution of Whispec can systematically enumerate inputs that maximize code coverage. Experiments using a variety of data structure implementations from the Java libraries show that our framework generates significantly smaller test suites (while maximizing coverage) than those generated by previous specification-based approaches.