Testing-Based Formal Verification for Theorems and Its Application in Software Specification Verification

Verifying a specification for software can be converted into theorem proving. In this paper, we describe a testing-based formal verification (TBFV) method for automatically testing theorems. The advantage of the method over conventional theorem proving is that it can quickly detect faults if the theorem is not valid and quickly provide us with confidence in the validity of the theorem if it is valid. We discuss the principle and algorithms for test case generation in TBFV and present an example to illustrate how TBFV can be applied in checking designs. We also present a prototype supporting tool we have developed and a controlled experiment for evaluating the performance of TBFV. The result shows that TBFV is effective and efficient to find faults in certain setting.

[1]  Bernhard K. Aichernig,et al.  Test case generation by OCL mutation and constraint solving , 2005, Fifth International Conference on Quality Software (QSIC'05).

[2]  Peter Dybjer,et al.  Combining Testing and Proving in Dependent Type Theory , 2003, TPHOLs.

[3]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[4]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[5]  Christoph Benzmüller,et al.  Preface: Proceedings of the 8th Workshop on User Interfaces for Theorem Provers (UITP 2008) , 2009 .

[6]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[7]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[8]  Reiner Hähnle,et al.  Using a Software Testing Technique to Improve Theorem Proving , 2003, FATES.

[9]  Natarajan Shankar,et al.  Formal Verification for Fault-Tolerant Architectures: Prolegomena to the Design of PVS , 1995, IEEE Trans. Software Eng..

[10]  Panagiotis Manolios,et al.  Automated specification analysis using an interactive theorem prover , 2011, 2011 Formal Methods in Computer-Aided Design (FMCAD).

[11]  Benjamin C. Pierce,et al.  Foundational Property-Based Testing , 2015, ITP.

[12]  Panagiotis Manolios,et al.  Integrating Testing and Interactive Theorem Proving , 2011, ACL2.

[13]  Boris Beizer,et al.  Black-box testing , 1995 .

[14]  Shaoying Liu.  Formal Engineering for Industrial Software Development: Using the SOFL Method , 2004 .

[15]  Thomas Santen,et al.  Automating Test Case Generation from Z Specifications with Isabelle , 1997, ZUM.

[16]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[17]  Panagiotis Manolios,et al.  ACL2s: "The ACL2 Sedan" , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[18]  Shaoying Liu,et al.  A Rigorous Method for Inspection of Model-Based Formal Specifications , 2010, IEEE Transactions on Reliability.

[19]  Bernhard K. Aichernig.  Model-Based Mutation Testing of Reactive Systems - From Semantics to Automated Test-Case Generation , 2013, Theories of Programming and Formal Methods.

[20]  Bruno Dutertre,et al.  Yices 2.2 , 2014, CAV.

[21]  Shaoying Liu,et al.  A Decompositional Approach to Automatic Test Case Generation Based on Formal Specifications , 2010, 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement.

[22]  Sarfraz Khurshid,et al.  A Case for Efficient Solution Enumeration , 2003, SAT.

[23]  Gary T. Leavens,et al.  Specification and verification challenges for sequential object-oriented programs , 2007, Formal Aspects of Computing.

[24]  Lukas Bulwahn,et al.  The New Quickcheck for Isabelle - Random, Exhaustive and Symbolic Testing under One Roof , 2012, CPP.

[25]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .