Counterexample generation for higher-order logic using functional and logic programming

This thesis presents a counterexample generator for the interactive theorem prover Isabelle/HOL that uncovers faulty specifications and invalid conjectures using testing methods. Its contributions are two testing strategies: exhaustive testing with concrete values; and symbolic testing, evaluating conjectures with a narrowing strategy. We present techniques to deal with conditional conjectures, including an approach to synthesize test data generators derived from the premise's definition.

[1]  Simon L. Peyton Jones,et al.  Template meta-programming for Haskell , 2002, Haskell '02.

[2]  Sebastian Fischer,et al.  Purely functional lazy non-deterministic programming , 2009, Journal of Functional Programming.

[3]  Arnaud Gotlieb,et al.  A First Step in the Design of a Formally Verified Constraint-Based Testing Tool: FocalTest , 2012, TAP@TOOLS.

[4]  Simon L. Peyton Jones.  Haskell 98: Standard Prelude , 2003, J. Funct. Program..

[5]  René Thiemann,et al.  Executable Transitive Closures of Finite Relations , 2011, Arch. Formal Proofs.

[6]  Sebastian Mödersheim,et al.  The AVISPA Tool for the Automated Validation of Internet Security Protocols and Applications , 2005, CAV.

[7]  Colin Runciman,et al.  Finding Inputs that Reach a Target Expression , 2007, Seventh IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2007).

[8]  Brian Huffman,et al.  Lifting and Transfer: A Modular Design for Quotients in Isabelle/HOL , 2013, CPP.

[9]  Simon L. Peyton Jones,et al.  A semantics for imprecise exceptions , 1999, PLDI '99.

[10]  Philip Wadler,et al.  How to make ad-hoc polymorphism less ad hoc , 1989, POPL '89.

[11]  Laurent Fribourg,et al.  SLOG: A Logic Programming Language Interpreter Based on Clausal Superposition and Rewriting , 1985, SLP.

[12]  Lee Naish,et al.  Adding equations to NU-Prolog , 1991, PLILP.

[13]  Tobias Nipkow,et al.  Proof Terms for Simply Typed Higher Order Logic , 2000, TPHOLs.

[14]  Koen Claessen,et al.  Testing Polymorphic Properties , 2010, ESOP.

[15]  Wolfgang Lux.  Implementing Encapsulated Search for a Lazy Functional Logic Language , 1999, Fuji International Symposium on Functional and Logic Programming.

[16]  Panagiotis Manolios,et al.  Computer-Aided Reasoning: An Approach , 2011 .

[17]  M. Gordon,et al.  Introduction to HOL: a theorem proving environment for higher order logic , 1993 .

[18]  Chandrasekhar Boyapati,et al.  Efficient software model checking of data structure properties , 2006, OOPSLA '06.

[19]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[20]  Rex L. Page.  Property-Based Testing and Verification: A Catalog of Classroom Examples , 2011, IFL.

[21]  Bruno Blanchet,et al.  An efficient cryptographic protocol verifier based on prolog rules , 2001, Proceedings. 14th IEEE Computer Security Foundations Workshop, 2001..

[22]  John Harrison,et al.  HOL Light: A Tutorial Introduction , 1996, FMCAD.

[23]  Panagiotis Manolios,et al.  Integrating Testing and Interactive Theorem Proving , 2011, ACL2.

[24]  Philip Wadler,et al.  Packrat parsing:: simple, powerful, lazy, linear time, functional pearl , 2002, ICFP '02.

[25]  Alexis Darrasse,et al.  Fast and sound random generation for automated testing and benchmarking in objective Caml , 2009, ML '09.

[26]  Sergio Antoy,et al.  The Pull-Tab Transformation , 2010 .

[27]  Tobias Nipkow,et al.  A Proof Assistant for Higher-Order Logic , 2002 .

[28]  Tobias Nipkow,et al.  Code Generation via Higher-Order Rewrite Systems , 2010, FLOPS.

[29]  Sebastian Fischer.  Reinventing Haskell Backtracking , 2009, GI Jahrestagung.

[30]  Herbert Kuchen,et al.  Systematic generation of glass-box test cases for functional logic programs , 2007, PPDP '07.

[31]  Emina Torlak,et al.  Kodkod: A Relational Model Finder , 2007, TACAS.

[32]  Tjark Weber,et al.  Bounded Model Generation for Isabelle/HOL , 2005, D/PDPAR@IJCAR.

[33]  Tobias Nipkow,et al.  Term rewriting and all that , 1998 .

[34]  William McCune,et al.  Mace4 Reference Manual and Guide , 2003, ArXiv.

[35]  Carl Eastlund.  DoubleCheck your theorems , 2009, ACL2 '09.

[36]  Jean-Marie Hullot,et al.  Canonical Forms and Unification , 1980, CADE.

[37]  Alonzo Church,et al.  A formulation of the simple theory of types , 1940, Journal of Symbolic Logic.

[38]  Jasmin Christian Blanchette,et al.  Monotonicity Inference for Higher-Order Formulas , 2011, Journal of Automated Reasoning.

[39]  Alastair F. Donaldson,et al.  Software Model Checking , 2014, Computing Handbook, 3rd ed..

[40]  Jacques D. Fleuriot,et al.  IsaPlanner: A Prototype Proof Planner in Isabelle , 2003, CADE.

[41]  Uday S. Reddy,et al.  Narrowing as the Operational Semantics of Functional Languages , 1985, SLP.

[42]  Lawrence C. Paulson,et al.  ML for the working programmer (2. ed.) , 1996 .

[43]  Corrado Moiso,et al.  Kernel-LEAF: A Logic plus Functional Language , 1991, J. Comput. Syst. Sci..

[44]  Tobias Nipkow.  Verifying a Hotel Key Card System , 2006, ICTAC.

[45]  Giampaolo Bella,et al.  Inductive verification of cryptographic protocols , 2000 .

[46]  Andreas Lochbihler.  Jinja with Threads , 2007, Arch. Formal Proofs.

[47]  Louis Morgan.  Random Testing of ML Programs , 2010 .

[48]  Markus Wenzel,et al.  Type Classes and Overloading in Higher-Order Logic , 1997, TPHOLs.

[49]  Tobias Nipkow,et al.  A Compiled Implementation of Normalization by Evaluation , 2008, TPHOLs.

[50]  Gavin Lowe.  Casper: a compiler for the analysis of security protocols , 1998 .

[51]  Steffen Hölldobler,et al.  Foundations of Equational Logic Programming , 1989, Lecture Notes in Computer Science.

[52]  Stefan Berghofer,et al.  Turning Inductive into Equational Specifications , 2009, TPHOLs.

[53]  Ralf Hinze,et al.  Comparing Approaches to Generic Programming in Haskell , 2006, SSDGP.

[54]  Lars-Åke Fredlund,et al.  Property-Based Testing - The ProTest Project , 2009, FMCO.

[55]  Ulf Norell,et al.  Using temporal relations to specify and test an instant messaging server , 2010, AST '10.

[56]  David Maier,et al.  Magic sets and other strange ways to implement logic programs (extended abstract) , 1985, PODS '86.

[57]  Chandrasekhar Boyapati,et al.  Efficient software model checking of soundness of type systems , 2008, OOPSLA.

[58]  Sebastian Fischer,et al.  EasyCheck - Test Data for Free , 2008, FLOPS.

[59]  Tobias Nipkow,et al.  The Isabelle Framework , 2008, TPHOLs.

[60]  Stefan Berghofer,et al.  Inductive Datatypes in HOL - Lessons Learned in Formal-Logic Engineering , 1999, TPHOLs.

[61]  Hantao Zhang,et al.  SEM: a System for Enumerating Models , 1995, IJCAI.

[62]  Meng Wang,et al.  Feat: functional enumeration of algebraic types , 2012, Haskell.

[63]  Tjark Weber,et al.  SAT-based finite model generation for higher-order logic , 2008 .

[64]  John M. Hughes,et al.  Testing a database for race conditions with QuickCheck: none , 2011, Erlang '11.

[65]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[66]  Andrei Popescu,et al.  Foundational, Compositional (Co)datatypes for Higher-Order Logic: Category Theory Applied to Theorem Proving , 2012, 2012 27th Annual IEEE Symposium on Logic in Computer Science.

[67]  Lawrence C. Paulson,et al.  The Inductive Approach to Verifying Cryptographic Protocols , 2021, J. Comput. Secur..

[68]  Ralf Hinze,et al.  Generic Haskell: Practice and Theory , 2003, Generic Programming.

[69]  Tobias Nipkow,et al.  Nitpick: A Counterexample Generator for Higher-Order Logic Based on a Relational Model Finder , 2010, ITP.

[70]  Colin Runciman,et al.  Smallcheck and lazy smallcheck: automatic exhaustive testing for small values , 2008, Haskell '08.

[71]  François Pessaux,et al.  Trusted Software within Focal , 2008 .

[72]  Lawrence Charles Paulson,et al.  Isabelle: A Generic Theorem Prover , 1994 .

[73]  Simon L. Peyton Jones,et al.  Report on the programming language Haskell: a non-strict, purely functional language version 1.2 , 1992, SIGP.

[74]  Colin Runciman,et al.  Lazy Generation of Canonical Test Programs , 2011, IFL.

[75]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[76]  Clemens Ballarin,et al.  Interpretation of Locales in Isabelle: Theories and Proof Contexts , 2006, MKM.

[77]  Fredrik Lindblad.  Property Directed Generation of First-Order Test Data , 2007, Trends in Functional Programming.

[78]  Rita Loogen,et al.  Lazy Narrowing in a Graph Machine , 1990, ALP.

[79]  James Cheney,et al.  Mechanized metatheory model-checking , 2007, PPDP '07.

[80]  Florian Haftmann,et al.  Code generation from specifications in higher-order logic , 2009 .

[81]  Koen Claessen,et al.  Finding race conditions in Erlang with QuickCheck and PULSE , 2009, ICFP.

[82]  Catherine Dubois,et al.  Functional Testing in the Focal Environment , 2008, TAP.

[83]  S. Owre.  Random Testing in PVS , 2006 .

[84]  Danny Dolev,et al.  On the security of public key protocols , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[85]  J. Michael Spivey,et al.  Combinators for breadth-first search , 2000, Journal of Functional Programming.

[86]  Lawrence Charles Paulson,et al.  ML for the working programmer , 1991 .

[87]  Marinus J. Plasmeijer,et al.  Gast: Generic Automated Software Testing , 2002, IFL.

[88]  Philip Wadler.  Monads for Functional Programming , 1995, Advanced Functional Programming.

[89]  David Scott Warren,et al.  XSB: Extending Prolog with Tabled Logic Programming , 2010, Theory and Practice of Logic Programming.

[90]  Sebastian Fischer,et al.  Transforming Functional Logic Programs into Monadic Functional Programs , 2010, WFLP.

[91]  Natarajan Shankar,et al.  PVS: A Prototype Verification System , 1992, CADE.

[92]  Peter Dybjer,et al.  Combining Testing and Proving in Dependent Type Theory , 2003, TPHOLs.

[93]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .

[94]  Roger M. Needham,et al.  Using encryption for authentication in large networks of computers , 1978, CACM.

[95]  Tobias Nipkow,et al.  Random testing in Isabelle/HOL , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[96]  Dale A. Miller,et al.  AN OVERVIEW OF PROLOG , 1988 .

[97]  Vesa A. J. Karvonen.  Generics for the working ML'er , 2007, ML '07.

[98]  Wim Vanhoof,et al.  Automatic Generation of Test Inputs for Mercury , 2008, LOPSTR.

[99]  Christian Urban,et al.  alpha-Prolog: A Logic Programming Language with Names, Binding and a-Equivalence , 2004, ICLP.

[100]  Sebastian Fischer,et al.  On functional logic programming and its application to testing , 2010 .

[101]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[102]  Alan Bundy,et al.  Conjecture Synthesis for Inductive Theories , 2011, Journal of Automated Reasoning.

[103]  Céline Rouveirol.  Flattening and Saturation: Two Representation Changes for Generalization , 2004, Machine Learning.

[104]  Ralf Hinze,et al.  Deriving backtracking monad transformers , 2000, ICFP '00.

[105]  Michael Hanus,et al.  Functional logic programming , 2010, CACM.

[106]  Michael Hanus,et al.  KiCS2: A New Compiler from Curry to Haskell , 2011, WFLP.

[107]  Andy King,et al.  Mode Analysis Domains for Typed Logic Programs , 1999, LOPSTR.

[108]  Gavin Lowe,et al.  An Attack on the Needham-Schroeder Public-Key Authentication Protocol , 1995, Inf. Process. Lett..

[109]  Koen Claessen,et al.  QuickSpec: Guessing Formal Specifications Using Testing , 2010, TAP@TOOLS.

[110]  John Hughes,et al.  Testing telecoms software with quviq QuickCheck , 2006, ERLANG '06.

[111]  Andreas Lochbihler.  Formalising FinFuns - Generating Code for Functions as Data from Isabelle/HOL , 2009, TPHOLs.

[112]  Arnaud Gotlieb,et al.  Constraint Reasoning in FocalTest , 2010, ICSOFT.

[113]  Peter J. Stuckey,et al.  Constraint-based mode analysis of mercury , 2002, PPDP '02.