The New Quickcheck for Isabelle - Random, Exhaustive and Symbolic Testing under One Roof

The new Quickcheck is a counterexample generator for Isabelle/HOL that uncovers faulty specifications and invalid conjectures using various testing strategies. The previous Quickcheck only tested conjectures by random testing. The new Quickcheck extends the previous one and integrates two novel testing strategies: exhaustive testing with concrete values; and symbolic testing, evaluating conjectures with a narrowing strategy. Orthogonally to the strategies, we address two general issues: First, we extend the class of executable conjectures and specifications, and second, we present techniques to deal with conditional conjectures, i.e., conjectures with premises. We evaluate the testing strategies and techniques on a number of specifications, functional data structures and a hotel key card system.

[1]  Philip Wadler,et al.  Packrat parsing:: simple, powerful, lazy, linear time, functional pearl , 2002, ICFP '02.

[2]  Markus Wenzel,et al.  Type Classes and Overloading in Higher-Order Logic , 1997, TPHOLs.

[3]  Tobias Nipkow.  Verifying a Hotel Key Card System , 2006, ICTAC.

[4]  Joyce L. Vedral,et al.  Functional Programming Languages and Computer Architecture , 1989, Lecture Notes in Computer Science.

[5]  Fredrik Lindblad.  Property Directed Generation of First-Order Test Data , 2007, Trends in Functional Programming.

[6]  Tobias Nipkow,et al.  Code Generation via Higher-Order Rewrite Systems , 2010, FLOPS.

[7]  Peter Dybjer,et al.  Combining Testing and Proving in Dependent Type Theory , 2003, TPHOLs.

[8]  Panagiotis Manolios,et al.  Integrating Testing and Interactive Theorem Proving , 2011, ACL2.

[9]  Tjark Weber,et al.  Bounded Model Generation for Isabelle/HOL , 2005, D/PDPAR@IJCAR.

[10]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[11]  Frank Wolter,et al.  Monodic fragments of first-order temporal logics: 2000-2001 A.D , 2001, LPAR.

[12]  Susmit Sarkar,et al.  Nitpicking c++ concurrency , 2011, PPDP.

[13]  Sebastian Fischer,et al.  EasyCheck - Test Data for Free , 2008, FLOPS.

[14]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[15]  Tobias Nipkow,et al.  Automatic Proof and Disproof in Isabelle/HOL , 2011, FroCoS.

[16]  Tobias Nipkow,et al.  Random testing in Isabelle/HOL , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[17]  Tobias Nipkow,et al.  A Proof Assistant for Higher-Order Logic , 2002 .

[18]  Kamel Barkaoui,et al.  Theoretical Aspects of Computing - ICTAC 2006, Third International Colloquium, Tunis, Tunisia, November 20-24, 2006, Proceedings , 2006, ICTAC.

[19]  Carl Eastlund.  DoubleCheck your theorems , 2009, ACL2 '09.

[20]  Michael Hanus,et al.  Functional logic programming , 2010, CACM.

[21]  Lukas Bulwahn,et al.  Smart Testing of Functional Programs in Isabelle , 2012, LPAR.

[22]  Tjark Weber,et al.  SAT-based finite model generation for higher-order logic , 2008 .

[23]  Colin Runciman,et al.  Smallcheck and lazy smallcheck: automatic exhaustive testing for small values , 2008, Haskell '08.

[24]  Sebastian Fischer,et al.  Purely functional lazy non-deterministic programming , 2009, Journal of Functional Programming.

[25]  David Aspinall,et al.  Formalising Java's Data Race Free Guarantee , 2007, TPHOLs.

[26]  Tom Schrijvers,et al.  Functional and Logic Programming , 2012, Lecture Notes in Computer Science.

[27]  S. Owre.  Random Testing in PVS , 2006 .

[28]  Tobias Nipkow,et al.  Nitpick: A Counterexample Generator for Higher-Order Logic Based on a Relational Model Finder , 2010, ITP.