QuickChick: Property-based testing for Coq

Designing complex systems that provide strong safety and security guarantees is challenging (e.g. programming languages, language compilers and runtimes, reference monitors, operating systems, hardware, etc). Proof assistants such as Coq (The Coq team, 1984-now) are invaluable for showing formally that such systems indeed satisfy the properties intended by their designers. However, carrying out formal proofs while designing even a relatively simple system can be an exercise in frustration, with a great deal of time spent attempting to prove things about broken definitions, and countless iterations for discovering the correct lemmas and strengthening inductive invariants. The long-term goal of this project1 is to reduce the cost of producing formally verified systems by integrating property-based testing (PBT) with the Coq proof assistant. Ideally, our solution will achieve the best of testing and proving, by producing easily understandable counterexamples and guiding users towards correct system designs and corresponding formal evidence of their correctness. The use of PBT will dramatically decrease the number of failed proof attempts in Coq developments by allowing users to find errors in definitions and conjectured properties early in the design process, and to postpone verification attempts until they are reasonably confident that their system is correct. PBT will also help during the verification process by quickly validating proof goals, potential lemmas, and inductive invariants. Our solution will provide automation of common patterns, yet keep the user fully in control. These improvements will lower the barrier to entry and increase adoption of the Coq proof assistant. Moreover, integrating PBT with Coq will provide an easier path going from systematic testing to formal verification, by encouraging developers to write specifications that can be only tested at first and later formally verified. It will also allow PBT users to verify that they are testing the right properties and to evaluate the thoroughness of their testing. Achieving all this requires improvements to the state-of-the-art both in PBT and formal verification research, which we discuss in the next section. While property-based testing has already been integrated with relative success into other proof assistants such as Isabelle (Bulwahn, 2013) and ACL2 (Chamarthi et al., 2011), the logic of Coq is much richer, which raises additional challenges. Also, these previous efforts were aimed at full automation, leaving no space for user customization or interaction, which is, in our experience, crucial for thorough testing that finds interesting bugs and drives the design and verification of nontrivial systems. As a necessary first step towards the final goal of this project we have ported the QuickCheck framework (Claessen and Hughes, 2000; Hughes, 2007) from Haskell to Coq, producing an prototype Coq plugin