On theorem prover-based testing

HOL-TestGen is a specification and test case generation environment extending the interactive theorem prover Isabelle/HOL. As such, Testgen allows for an integrated workflow supporting interactive theorem proving, test case generation, and test data generation. The HOL-TestGen method is two-staged: first, the original formula is partitioned into test cases by transformation into a normal form called test theorem. Second, the test cases are analyzed for ground instances (the test data) satisfying the constraints of the test cases. Particular emphasis is put on the control of explicit test-hypotheses which can be proven over concrete programs. Due to the generality of the underlying framework, our system can be used for black-box unit, sequence, reactive sequence and white-box test scenarios. Although based on particularly clean theoretical foundations, the system can be applied for substantial case-studies.

[1]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.

[2]  Achim D. Brucker,et al.  Interactive Testing with HOL-TestGen , 2005, FATES.

[3]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[4]  Andrew William Roscoe,et al.  The Theory and Practice of Concurrency , 1997 .

[5]  Alain Denise,et al.  Coverage-biased Random Exploration of Models , 2008, Electron. Notes Theor. Comput. Sci..

[6]  Bruno Legeard,et al.  LEIRIOS Test Generator: Automated Test Generation from B Models , 2007, B.

[7]  Achim D. Brucker,et al.  Test-Sequence Generation with Hol-TestGen with an Application to Firewall Testing , 2007, TAP.

[8]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[9]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[10]  N. Falconer.  Structured Programming , 1973, Nature.

[11]  Marius Mikucionis,et al.  Formal Methods and Testing , 2008 .

[12]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[13]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.

[14]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[15]  Burkhart Wolff,et al.  A Corrected Failure Divergence Model for CSP in Isabelle/HOL , 1997, FME.

[16]  Tobias Nipkow,et al.  Winskel is (almost) Right: Towards a Mechanized Semantics Textbook , 1996, Formal Aspects of Computing.

[17]  Bruno Marre,et al.  Test Selection Strategies for Lustre Descriptions in GATeL , 2005, Electron. Notes Theor. Comput. Sci..

[18]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[19]  Kenneth A. Foster.  Error Sensitive Test Cases Analysis (ESTCA) , 1980, IEEE Transactions on Software Engineering.

[20]  Eugenio Moggi,et al.  Notions of Computation and Monads , 1991, Inf. Comput..

[21]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[22]  Lawrence Charles Paulson,et al.  Isabelle/HOL: A Proof Assistant for Higher-Order Logic , 2002 .

[23]  Margus Veanes,et al.  Instrumenting scenarios in a model-driven development environment , 2004, Inf. Softw. Technol..

[24]  Harry D. Raduege,et al.  Securing Cyberspace for the 44th Presidency , 2008 .

[25]  David von Oheimb.  Analyzing Java in Isabelle-HOL: formalization, type safety and Hoare logic , 2001 .

[26]  G. Winskel.  The formal semantics of programming languages , 1993 .

[27]  Peter B. Andrews.  An introduction to mathematical logic and type theory - to truth through proof , 1986, Computer science and applied mathematics.

[28]  Tobias Nipkow.  Winskel is (Almost) Right: Towards a Mechanized Semantics Textbook , 1996, FSTTCS.

[29]  Achim D. Brucker,et al.  Symbolic Test Case Generation for Primitive Recursive Functions , 2004, FATES.

[30]  Tobias Nipkow,et al.  Random testing in Isabelle/HOL , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[31]  Philip Wadler.  XQuery: A Typed Functional Language for Querying XML , 2002, Advanced Functional Programming.

[32]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[33]  I. C. Wand,et al.  A.P.I.C. Studies in Data Processing , 1989 .

[34]  David,et al.  Analyzing Java in Isabelle/HOL , 2001 .

[35]  Fatiha Zaïdi,et al.  A Formal Framework for Service Orchestration Testing Based on Symbolic Transition Systems , 2009, TestCom/FATES.

[36]  Achim D. Brucker,et al.  Verified Firewall Policy Transformations for Test Case Generation , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[37]  Achim D. Brucker,et al.  An approach to modular and testable security models of real-world health-care applications , 2011, SACMAT '11.

[38]  Markus Wenzel,et al.  Isabelle, Isar - a versatile environment for human readable formal proof documents , 2002 .

[39]  Nikolaj Bjørner,et al.  Path Feasibility Analysis for String-Manipulating Programs , 2009, TACAS.

[40]  Achim D. Brucker,et al.  hol-TestGen , 2009, FASE.

[41]  D. Gabbay,et al.  Handbook of tableau methods , 1999 .

[42]  Alok Nandan,et al.  Model-Based Quality Assurance of Windows Protocol Documentation , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[43]  Achim D. Brucker,et al.  HOL-TestGen An Interactive Test-case Generation Framework , 2009 .

[44]  Glynn Winskel,et al.  The formal semantics of programming languages - an introduction , 1993, Foundation of computing series.

[45]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[46]  Christoph Lange,et al.  Flyspeck in a Semantic Wiki , 2008, SemWiki.

[47]  Antti Huima,et al.  Implementing Conformiq Qtronic , 2007, TestCom/FATES.

[48]  Lawrence C. Paulson,et al.  A Generic Tableau Prover and its Integration with Isabelle , 1999, J. Univers. Comput. Sci..

[49]  Corina S. Pasareanu,et al.  JPF-SE: A Symbolic Execution Extension to Java PathFinder , 2007, TACAS.

[50]  David Aspinall,et al.  Formalising Java's Data Race Free Guarantee , 2007, TPHOLs.

[51]  Nikolai Tillmann,et al.  Parameterized Unit Testing with Pex , 2008, TAP.

[52]  Achim D. Brucker,et al.  Model-Based Firewall Conformance Testing , 2008, TestCom/FATES.

[53]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[54]  Tobias Nipkow,et al.  A Compiled Implementation of Normalization by Evaluation , 2008, TPHOLs.

[55]  Gerwin Klein,et al.  Operating system verification—An overview , 2009 .

[56]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[57]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[58]  Achim D. Brucker,et al.  An Extensible Encoding of Object-oriented Data Models in hol , 2008, Journal of Automated Reasoning.

[59]  Philip Wadler,et al.  Monads for functional programming , 1995, NATO ASI PDC.

[60]  Mogens Nielsen,et al.  TAPSOFT '95: Theory and Practice of Software Development , 1995, Lecture Notes in Computer Science.

[61]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[62]  Gregory Tassey,et al.  Prepared for what , 2007 .

[63]  Alonzo Church,et al.  A formulation of the simple theory of types , 1940, Journal of Symbolic Logic.