Generating good generators for inductive relations

Property-based random testing (PBRT) is widely used in the functional programming and verification communities. For testing simple properties, PBRT tools such as QuickCheck can automatically generate random inputs of a given type. But for more complex properties, effective testing often demands generators for random inputs that belong to a given type and satisfy some logical condition. QuickCheck provides a library of combinators for building such generators by hand, but this can be tedious for simple conditions and error prone for more complex ones. Fortunately, the process can often be automated. The most prominent method, narrowing, works by traversing the structure of the condition, lazily instantiating parts of the data structure as constraints involving them are met. We show how to use ideas from narrowing to compile a large subclass of Coq's inductive relations into efficient generators, avoiding the interpretive overhead of previous implementations. More importantly, the same compilation technique allows us to produce proof terms certifying that each derived generator is good---i.e., sound and complete with respect to the inductive relation it was derived from. We implement our algorithm as an extension of QuickChick, an existing tool for property-based testing in Coq. We evaluate our method by automatically deriving good generators for the majority of the specifications in Software Foundations, a formalized textbook on programming language foundations.

[1]  Herbert Kuchen,et al.  Systematic generation of glass-box test cases for functional logic programs , 2007, PPDP '07.

[2]  Sebastian Fischer,et al.  EasyCheck - Test Data for Free , 2008, FLOPS.

[3]  Benjamin C. Pierce,et al.  Testing Noninterference , Quickly ∗ ( short talk ) , 2013 .

[4]  Koen Claessen,et al.  Making Random Judgments: Automatically Generating Well-Typed Terms from the Definition of a Type-System , 2015, ESOP.

[5]  P. H. Testing an Optimising Compiler by Generating Random Lambda Terms , 2012 .

[6]  Matthieu Sozeau,et al.  First-Class Type Classes , 2008, TPHOLs.

[7]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[8]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[9]  M. Hanus,et al.  Curry: A Truly Functional Logic Language , 1995 .

[10]  Benjamin C. Pierce,et al.  Beginner's luck: a language for property-based generators , 2016, POPL.

[11]  Arnaud Gotlieb,et al.  FocalTest: A Constraint Programming Approach for Property-Based Testing , 2010, ICSOFT.

[12]  John Hughes,et al.  Testing Erlang data types with quviq quickcheck , 2008, ERLANG '08.

[13]  Lukas Bulwahn,et al.  Random, Exhaustive and Symbolic Testing under One Roof , 2012 .

[14]  Konstantinos Sagonas,et al.  A PropEr integration of types and function specifications with property-based testing , 2011, Erlang Workshop.

[15]  Colin Runciman,et al.  Smallcheck and lazy smallcheck: automatic exhaustive testing for small values , 2008, Haskell '08.

[16]  Benjamin C. Pierce,et al.  Testing noninterference, quickly , 2016, Journal of Functional Programming.

[17]  Lukas Bulwahn,et al.  The New Quickcheck for Isabelle - Random, Exhaustive and Symbolic Testing under One Roof , 2012, CPP.

[18]  David Delahaye,et al.  Producing Certified Functional Code from Inductive Specifications , 2012, CPP.

[19]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[20]  John Hughes,et al.  QuickCheck Testing for Fun and Profit , 2007, PADL.

[21]  Amir Pnueli,et al.  Translation Validation , 1998, TACAS.

[22]  Peter Dybjer,et al.  Combining Testing and Proving in Dependent Type Theory , 2003, TPHOLs.

[23]  Koen Claessen,et al.  Generating constrained random data with uniform distribution , 2014, Journal of Functional Programming.

[24]  Fredrik Lindblad Property Directed Generation of First-Order Test Data , 2007, Trends in Functional Programming.

[25]  Peter Dybjer,et al.  Verifying Haskell programs by combining testing, model checking and interactive theorem proving , 2004, Inf. Softw. Technol..

[26]  Viktor Kuncak,et al.  SciFe: Scala framework for efficient enumeration of data structures with invariants , 2014, SCALA@ECOOP.

[27]  Lukas Bulwahn,et al.  Smart Testing of Functional Programs in Isabelle , 2012, LPAR.

[28]  Benjamin C. Pierce,et al.  Foundational Property-Based Testing , 2015, ITP.

[29]  Panagiotis Manolios,et al.  Integrating Testing and Interactive Theorem Proving , 2011, ACL2.

[30]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[31]  B. Pierce,et al.  QuickChick: Property-based testing for Coq , 2014 .

[32]  David Delahaye,et al.  Extracting Purely Functional Contents from Logical Inductive Types , 2007, TPHOLs.

[33]  Viktor Kuncak,et al.  Programming with enumerable sets of structures , 2015, OOPSLA.

[34]  Ranjit Jhala,et al.  Type Targeted Testing , 2014, ESOP.

[35]  Rachid Echahed,et al.  A needed narrowing strategy , 2000, JACM.

[36]  Philip Wadler,et al.  How to make ad-hoc polymorphism less ad hoc , 1989, POPL '89.

[37]  Arnaud Gotlieb Euclide: A Constraint-Based Testing Framework for Critical C Programs , 2009, 2009 International Conference on Software Testing Verification and Validation.