Integrating formal verification and conformance testing for reactive systems

In this paper, we describe a methodology integrating verification and conformance testing. A specification of a system - an extended input-output automaton, which may be infinite-state - and a set of safety properties ("nothing bad ever happens") and possibility properties ("something good may happen") are assumed. The properties are first tentatively verified on the specification using automatic techniques based on approximated state-space exploration, which are sound, but, as a price to pay for automation, are not complete for the given class of properties. Because of this incompleteness and of state-space explosion, the verification may not succeed in proving or disproving the properties. However, even if verification did not succeed, the testing phase can proceed and provide useful information about the implementation. Test cases are automatically and symbolically generated from the specification and the properties and are executed on a black-box implementation of the system. The test execution may detect violations of conformance between implementation and specification; in addition, it may detect violation/satisfaction of the properties by the implementation and by the specification. In this sense, testing completes verification. The approach is illustrated on simple examples and on a bounded retransmission protocol.

[1]  Thierry Jéron,et al.  An Approach to Symbolic Test Generation , 2000, IFM.

[2]  Grigore Rosu,et al.  Synthesizing Monitors for Safety Properties , 2002, TACAS.

[3]  Thierry Jéron,et al.  Verification and test generation for the SSCOP protocol , 2000, Sci. Comput. Program..

[4]  Wei Ding,et al.  Using a model checker to test safety properties , 2001, Proceedings Seventh IEEE International Conference on Engineering of Complex Computer Systems.

[5]  Bertrand Jeannet,et al.  Symbolic Test Selection Based on Approximate Analysis , 2005, TACAS.

[6]  R. G. de Vries.  Towards Formal Test Purposes , 2001 .

[7]  Orna Kupferman,et al.  Model Checking of Safety Properties , 1999, Formal Methods Syst. Des..

[8]  Bengt Jonsson,et al.  Specifying and Generating Test Cases Using Observer Automata , 2004, FATES.

[9]  Mihalis Yannakakis,et al.  Black Box Checking , 1999, FORTE.

[10]  Thierry Jéron,et al.  Automatic Verification and Conformance Testing for Validating Safety Properties of Reactive Systems , 2005, FM.

[11]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[12]  Nicolae Goga,et al.  Formal Test Automation: A Simple Experiment , 1999, IWTCS.

[13]  Thierry Jéron,et al.  Symbolic Determinisation of Extended Automata , 2006, IFIP TCS.

[14]  Duncan Clarke,et al.  STG: A Symbolic Test Generation Tool , 2002, TACAS.

[15]  Jan Tretmans,et al.  A Formal Approach to Conformance Testing , 1993, Protocol Test Systems.

[16]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[17]  Robert E. Shostak,et al.  A Practical Decision Procedure for Arithmetic with Function Symbols , 1979, JACM.

[18]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[19]  Leonardo Mendonça de Moura,et al.  Generating efficient test sets with a model checker , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[20]  Frits W. Vaandrager,et al.  Proof-Checking a Data Link Protocol , 1994, TYPES.

[21]  Bertrand Jeannet,et al.  Dynamic Partitioning in Linear Relation Analysis: Application to the Verification of Reactive Systems , 2003, Formal Methods Syst. Des..

[22]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.

[23]  Insup Lee,et al.  A Temporal Logic Based Theory of Test Coverage and Generation , 2002, TACAS.

[24]  Bertrand Jeannet,et al.  From Safety Verification to Safety Testing , 2004, TestCom.

[25]  Elena Leroux,et al.  Symbolic Test Generation for Reactive Systems with Data. , 2004 .

[26]  JeannetB..  Dynamic Partitioning in Linear Relation Analysis , 2003 .

[27]  Jan Tretmans,et al.  Testing Concurrent Systems: A Formal Approach , 1999, CONCUR.

[28]  Thierry Jéron,et al.  A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2005 .

[29]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.