Assertion-Driven Development: Assessing the Quality of Contracts Using Meta-Mutations

Agile development methods have gained momentum inthe last few years and, as a consequence, test-driven developmenthas become more prevalent in practice. However, test cases arenot sufficient for producing dependable software and we ratheradvocate approaches that emphasize the use of assertions orcontracts over that of test cases. Yet, writing self-checks in codehas been shown to be difficult and is itself prone to errors. Astandard technique to specify runtime properties is design-bycontract(DbC). But how can one test if the contracts themselvesare sensible and sufficient? We propose a measure to quantifythe goodness of contracts (or assertions in a broader sense). Weintroduce meta-mutations at the source code level to simulatecommon programmer errors that the self-checks are supposedto detect. We then use random mutation testing to determinea lower and upper bound on the detectable mutations andcompare these bounds with the number of mutants detected bythe contracts. Contracts are considered “good” if they detect acertain percentage of the detectable mutations.We have evaluatedour tools on Java classes with contracts specified using theJava Modeling Language (JML). We have additionally tested thecontract quality of 19 implementations, written independently bystudents, based on the same specification.