Semantics-Based Testing for Circus

The work presented in this thesis is a contribution to formal specification and verification methods. Formal specifications are used to describe a software, or more generally a system, in a mathematical unambiguous way. Formal verification techniques are defined on the basis of these specifications to ensure the correctness of the resulting system. However, formal methods are often not convenient and easy to use in real system developments. One of the reasons is that many specification formalisms are not rich enough to cover both data-oriented and behavioral requirements. Some specification languages were proposed to cover this kind of requirements. The Circus language distinguishes itself among these languages by a rich syntax and a fully integrated semantics.The aim of this thesis is to provide a formal environment for specifying and verifying complex systems. Specifications are written in Circus and verification is performed either by testing or by theorem proving. Similar specifications and verification environment have already been proposed. A specificity of our approach is to combine supports for proofs and test generation. Moreover, most test generation methods are based on a syntactic characterization of the studied languages. Our proposed environment is different since it is based on the denotational and operational semantics of Circus. The Isabelle/HOL theorem prover is the formal platform on top of which we built our specification and verification environment.The first main contribution of our work is the Isabelle/Circus specification and proof environment based on the denotational semantics of Circus. On top of Isabelle/HOL we provide a machine-checked shallow embedding of UTP, the semantics basis of Circus. This embedding is used to formalize the denotational semantics of the Circus language. The Isabelle/Circus environment associates to this semantics some parsing facilities that help writing Circus specifications. The proof support of Isabelle/HOL can be used directly to reason on these specifications thanks to the shallow embedding of the semantics. We present an application of the environment to refinement proofs on Circus processes (involving both data and behavioral aspects). The second main contribution is the CirTA testing framework build on top of Isabelle/Circus. The framework provides two symbolic test generation tactics that allow checking two notions of refinement: traces inclusion and deadlocks reduction. The framework is based on a shallow symbolic formalization of the operational semantics of Circus using Isabelle/Circus. Several symbolic definition and test generation tactics are defined in the CirTA framework. The formal infrastructure allows us to represent explicitly test theories as well as test selection hypothesis. Proof techniques and symbolic computations are the basis of test generation tactics. The test generation environment was used for a case study to test an existing message monitoring system. A specification of the system is written in Circus, and used to generate tests following the defined conformance relations. The tests are then compiled in forms of JUnit test methods and executed against a Java implementation of the monitoring system.This thesis is a step towards, on one hand, the development of sophisticated testing tools making use of proof techniques and, on the other hand, the integration of testing and proving within formally verified software developments.

[1]  J. Michael Spivey,et al.  Understanding Z : A specification language and its formal semantics , 1985, Cambridge tracts in theoretical computer science.

[2]  Cliff B. Jones,et al.  Systematic software development using VDM , 1986, Prentice Hall International Series in Computer Science.

[3]  Robert M. Hierons Comparing test sets and criteria in the presence of test hypotheses and fault domains , 2002, TSEM.

[4]  Jim Woodcock,et al.  The Semantics of Circus , 2002, ZB.

[5]  Steve A. Schneider Abstraction and Testing , 1999, World Congress on Formal Methods.

[6]  Steve Schneider The B-method - an introduction , 2001, The cornerstones of computing series.

[7]  J. Woodcock,et al.  Refinement in Circus , 2002, FME.

[8]  Rocco De Nicola,et al.  Testing Equivalences for Processes , 1984, Theor. Comput. Sci..

[9]  Edsger W. Dijkstra,et al.  A Discipline of Programming , 1976 .

[10]  Daniel Le Métayer,et al.  CASTING: a formally based software test generation method , 1997, First IEEE International Conference on Formal Engineering Methods.

[11]  Lawrence Charles Paulson,et al.  Isabelle/HOL: A Proof Assistant for Higher-Order Logic , 2002 .

[12]  Hartmut Ehrig,et al.  ACT ONE - An Algebraic Specification Language with two Levels of Semantics , 1983, ADT.

[13]  Laurent Fribourg,et al.  Test sets generation from algebraic specifications using logic programming , 1986, J. Syst. Softw..

[14]  Pascale Le Gall,et al.  Testing Data Types Implementations from Algebraic Specifications , 2008, Formal Methods and Testing.

[15]  Tobias Nipkow,et al.  Automatic Proof and Disproof in Isabelle/HOL , 2011, FroCoS.

[16]  Ana Cavalcanti,et al.  Testing for Refinement in CSP , 2007, ICFEM.

[17]  Steve A. Schneider,et al.  CSP theorems for communicating B machines , 2005, Formal Aspects of Computing.

[18]  Paul Ammann,et al.  Using Z specifications in category partition testing , 1992, COMPASS `92 Proceedings of the Seventh Annual Conference on Computer Assurance.

[19]  Paul Ammann,et al.  Using formal methods to derive test frames in category-partition testing , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[20]  Tommaso Bolognesi,et al.  Tableau methods to describe strong bisimilarity on LOTOS processes involving pure interleaving and enabling , 1994, FORTE.

[21]  P.A.V. Hall,et al.  Towards testing with respect to formal specification , 1988 .

[22]  Burkhart Wolff,et al.  Unifying Theories in Isabelle/HOL , 2010, UTP.

[23]  Augusto Sampaio,et al.  Guided Test Generation from CSP Models , 2008, ICTAC.

[24]  Thomas Santen,et al.  Automating Test Case Generation from Z Specifications with Isabelle , 1997, ZUM.

[25]  Bill Stoddart,et al.  An operational semantics for ZCCS , 1997, First IEEE International Conference on Formal Engineering Methods.

[26]  David A. Carrington,et al.  A Tale of Two Paradigms: Formal Methods and Software Testing , 1994, Z User Workshop.

[27]  Robert M. Hierons Verdict functions in testing with a fault domain or test hypotheses , 2009, TSEM.

[28]  Richard G. Hamlet,et al.  Data Abstraction, Implementation, Specification, and Testing , 1981, TOPL.

[29]  Hasan Ural,et al.  Data Flow Oriented Test Selection for Lotos , 1995, Comput. Networks ISDN Syst..

[30]  Robert M. Hierons,et al.  Testing a system specified using Statecharts and Z , 2001, Inf. Softw. Technol..

[31]  J. Christian Attiogbé,et al.  Specification of an access control system with a formalism combining CCS and CASL , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[32]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[33]  Andy J. Galloway Integrated formal methods with richer methodological profiles for the development of multi-perspective systems , 1996 .

[34]  Andrew William Roscoe,et al.  The Theory and Practice of Concurrency , 1997 .

[35]  Burkhart Wolff,et al.  Isabelle/Circus: A Process Specification and Verification Environment , 2012, VSTTE.

[36]  Marcel Vinícius Medeiros Oliveira Formal derivation of state-rich reactive programs using Circus , 2005 .

[37]  Marie-Claude Gaudel Software Testing Based on Formal Specification , 2007, PSSE.

[38]  Marie-Claude Gaudel,et al.  Testing from Formal Specifications, a Generic Approach , 2001, Ada-Europe.

[39]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[40]  Michael J. Butler,et al.  ProB: A Model Checker for B , 2003, FME.

[41]  Catherine Dubois,et al.  Why Would You Trust B ? , 2007, LPAR.

[42]  Jim Woodcock,et al.  Using Z - specification, refinement, and proof , 1996, Prentice Hall international series in computer science.

[43]  Alonzo Church,et al.  A formulation of the simple theory of types , 1940, Journal of Symbolic Logic.

[44]  Radu Mateescu,et al.  CADP 2006: A Toolbox for the Construction and Analysis of Distributed Processes , 2007, CAV.

[45]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[46]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[47]  Edward F. Moore,et al.  Gedanken-Experiments on Sequential Machines , 1956 .

[48]  Jan Peleska,et al.  Test Automation for Safety-Critical Systems: Industrial Application and Future Developments , 1996, FME.

[49]  Maximiliano Cristiá,et al.  Implementing and Applying the Stocks-Carrington Framework for Model-Based Testing , 2009, ICFEM.

[50]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[51]  Mirko Conrad,et al.  Test case design based on Z and the classification-tree method , 1997, First IEEE International Conference on Formal Engineering Methods.

[52]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[53]  J. McDermid,et al.  Testing , Proof and Automation . An Integrated Approach , 2000 .

[54]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[55]  Marie-Claude Gaudel,et al.  Testing Algebraic Data Types and Processes: A Unifying Theory , 1998, Formal Aspects of Computing.

[56]  Marius Mikucionis,et al.  Formal Methods and Testing , 2008 .

[57]  Behçet Sarikaya,et al.  Test Generation from LOTOS Specifications , 1991, IEEE Trans. Computers.

[58]  Steve A. Schneider,et al.  Concurrent and Real-time Systems: The CSP Approach , 1999 .

[59]  Marie-Claude Gaudel,et al.  Using algebraic specifications in software testing: A case study on the software of an automatic subway , 1993, J. Syst. Softw..

[60]  Ana Cavalcanti,et al.  Encoding Circus Programs in ProofPowerZ , 2008, UTP.

[61]  Jan Tretmans,et al.  A Symbolic Framework for Model-Based Testing , 2006, FATES/RV.

[62]  David A. Carrington,et al.  A Framework for Specification-Based Testing , 1996, IEEE Trans. Software Eng..

[63]  Jim Woodcock,et al.  A Tutorial Introduction to CSP in Unifying Theories of Programming , 2004, PSSE.

[64]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[65]  Marie-Claude Gaudel,et al.  Checking Models, Proving Programs, and Testing Systems , 2011, TAP@TOOLS.

[66]  Cécile Péraire,et al.  Test selection for object-oriented software based on formal specifications , 1998, PROCOMET.

[67]  Patrícia Duarte de Lima Machado Testing from Structured Algebraic Specifications , 2000, AMAST.

[68]  F. C. Hennie Fault detecting experiments for sequential circuits , 1964, SWCT.

[69]  Itu-T Specification and Description Language (SDL) , 1999 .

[70]  Jean-Raymond Abrial,et al.  The B-book - assigning programs to meanings , 1996 .

[71]  Ana Cavalcanti,et al.  Operational Semantics for Circus , 2008 .

[72]  Jan Tretmans,et al.  Testing Transition Systems: An Annotated Bibliography , 2000, MOVEP.

[73]  Bruno Marre,et al.  Test sequences generation from LUSTRE descriptions: GATEL , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[74]  Markus Roggenbach,et al.  CSP-CASL - A new integration of process algebra and algebraic specification , 2006, Theor. Comput. Sci..

[75]  Jin Song Dong,et al.  Blending Object-Z and Timed CSP: an introduction to TCOZ , 1998, Proceedings of the 20th International Conference on Software Engineering.

[76]  Mike Holcombe,et al.  Refinement in statechart testing , 2004, Softw. Test. Verification Reliab..

[77]  Jan Tretmans,et al.  Conformance Testing with Labelled Transition Systems: Implementation Relations and Test Generation , 1996, Comput. Networks ISDN Syst..

[78]  Augusto Sampaio,et al.  A Refinement Strategy for Circus , 2003, Formal Aspects of Computing.

[79]  Achim D. Brucker,et al.  On theorem prover-based testing , 2012, Formal Aspects of Computing.

[80]  Thierry Jéron,et al.  A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2005 .

[81]  Marie-Claude Gaudel,et al.  Testing processes from formal specifications with inputs, outputs and data types , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[82]  Ana Cavalcanti,et al.  Mechanical reasoning about families of UTP theories , 2009, Sci. Comput. Program..

[83]  Norbert Schirmer,et al.  State Spaces - The Locale Way , 2009, SSV.

[84]  Jim Woodcock,et al.  Unifying theories in ProofPower-Z , 2006, Formal Aspects of Computing.

[85]  Jim Woodcock,et al.  A Denotational Semantics for Circus , 2007, Refine@ICFEM.

[86]  Thierry Jéron,et al.  Automated test generation from SDL specifications , 1999, SDL Forum.

[87]  Achim D. Brucker,et al.  HOL-Z 2.0: A Proof Environment for Z-Specifications , 2003, J. Univers. Comput. Sci..

[88]  Radu Mateescu,et al.  CADP 2010: A Toolbox for the Construction and Analysis of Distributed Processes , 2011, TACAS.

[89]  Richard H. Carver,et al.  Reachability testing of concurrent programs , 2006, IEEE Transactions on Software Engineering.

[90]  Richard J. Boulton,et al.  Experience with Embedding Hardware Description Languages in HOL , 1992, TPCD.

[91]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[92]  Sebastian Burckhardt,et al.  Preemption Sealing for Efficient Concurrency Testing , 2010, TACAS.

[93]  C. A. R. Hoare,et al.  Communicating Sequential Processes (Reprint) , 1983, Commun. ACM.

[94]  Burkhart Wolff,et al.  A Corrected Failure Divergence Model for CSP in Isabelle/HOL , 1997, FME.

[95]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[96]  Robert M. Hierons Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[97]  Clemens Fischer CSP-OZ: a combination of object-Z and CSP , 1997 .

[98]  Marc Phalippou Relations d'implantation et hypothèses de test sur des automates à entrées et sorties , 1994 .

[99]  Ana Cavalcanti,et al.  Specification Coverage for Testing in Circus , 2010, UTP.

[100]  David Harel,et al.  Executable object modeling with statecharts , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[101]  Tobias Nipkow,et al.  Machine-Checking the Java Specification: Proving Type-Safety , 1999, Formal Syntax and Semantics of Java.

[102]  Roland Groz,et al.  Eight Years of Experience in Test Generation from FDTs using TVEDA , 1997, FORTE.

[103]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[104]  Peter D. Mosses,et al.  CASL: the Common Algebraic Specification Language , 2002, Theor. Comput. Sci..

[105]  Ana Cavalcanti,et al.  Testing for refinement in Circus , 2011, Acta Informatica.

[106]  Lawrence C. Paulson,et al.  The foundation of a generic theorem prover , 1989, Journal of Automated Reasoning.

[107]  Jan Peleska,et al.  Test automation of safety-critical reactive systems , 1997 .