A framework and tool support for the systematic testing of model-based specifications

Formal specifications can precisely and unambiguously define the required behavior of a software system or component. However, formal specifications are complex artifacts that need to be verified to ensure that they are consistent, complete, and validated against the requirements. Specification testing or animation tools exist to assist with this by allowing the specifier to interpret or execute the specification. However, currently little is known about how to do this effectively.This article presents a framework and tool support for the systematic testing of formal, model-based specifications. Several important generic properties that should be satisfied by model-based specifications are first identified. Following the idea of mutation analysis, we then use variants or mutants of the specification to check that these properties are satisfied. The framework also allows the specifier to test application-specific properties. All properties are tested for a range of states that are defined by the tester in the form of a testgraph, which is a directed graph that partially models the states and transitions of the specification being tested. Tool support is provided for the generation of the mutants, for automatically traversing the testgraph and executing the test cases, and for reporting any errors. The framework is demonstrated on a small specification and its application to three larger specifications is discussed. Experience indicates that the framework can be used effectively to test small to medium-sized specifications and that it can reveal a significant number of problems in these specifications.

[1]  D. J. Robson,et al.  The Testing of Object-Oriented Programs , 1993 .

[2]  S. Easterbrook,et al.  Generating Test Oracles via Model Checking , 1997 .

[3]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[4]  Neil J. Robinson,et al.  7.1.4 Automatic Generation and Verification of Design Specifications for Railway Signalling Applications , 2001 .

[5]  Shaoying Liu.  Verifying Consistency and Validity of Formal Specifications by Testing , 1999, World Congress on Formal Methods.

[6]  Michael Winikoff,et al.  Verifying Requirements Through Mathematical Modelling and Animation , 2000, Int. J. Softw. Eng. Knowl. Eng..

[7]  Hélène Waeselynck,et al.  B model animation for external verification , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[8]  David J. Robson,et al.  A State-Based Approach to the Testing of Class-Based Programs , 1995, Softw. Concepts Tools.

[9]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[10]  Colin O'Halloran,et al.  Experiences with PiZA, an Animator for Z , 1997, ZUM.

[11]  David A. Carrington,et al.  A Framework for Specification-Based Testing , 1996, IEEE Trans. Software Eng..

[12]  Roger Duke,et al.  Formal Object Oriented Specification Using Object-Z , 2000 .

[13]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[14]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[15]  David A. Carrington,et al.  A Tale of Two Paradigms: Formal Methods and Software Testing , 1994, Z User Workshop.

[16]  Leesa Murray,et al.  From Object-Z Specifications to ClassBench Test Suites , 2000, Softw. Test. Verification Reliab..

[17]  Constance L. Heitmeyer,et al.  SCR*: A Toolset for Specifying and Analyzing Software Requirements , 1998, CAV.

[18]  Leesa Murray,et al.  Formal Derivation of Finite State Machines for Class Testing , 1998, ZUM.

[19]  Mark Utting,et al.  Data structures for Z testing tools. , 2001 .

[20]  Paul A. Strooper,et al.  Possum: an animator for the SUM specification language , 1997, Proceedings of Joint 4th International Computer Science Conference and 4th Asia Pacific Software Engineering Conference.

[21]  Kai-Hsiung Chang,et al.  Testing object-oriented programs: from formal specification to test scenario generation , 1998, J. Syst. Softw..

[22]  Daniel Hoffman,et al.  The Testgraph Methodology: Automated Testing of Collection Classes , 1995, J. Object Oriented Program..

[23]  Nicholas A. Hamilton,et al.  A complete formal development using Cogito , 1998 .

[24]  MillerTim,et al.  A framework and tool support for the systematic testing of model-based specifications , 2003 .

[25]  Stephan Merz,et al.  Model Checking , 2000 .

[26]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[27]  HeitmeyerConstance,et al.  Using model checking to generate tests from requirements specifications , 1999 .

[28]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[29]  Paul A. Strooper,et al.  Requirements engineering and verification using specification animation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[30]  Robert M. Hierons.  Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[31]  Steve Schneider.  The B-method - an introduction , 2001, The cornerstones of computing series.

[32]  Tim Miller,et al.  Supporting the software testing process through specification animation , 2003, First International Conference onSoftware Engineering and Formal Methods, 2003.Proceedings..

[33]  Tim Miller,et al.  Animation can show only the presence of errors, never their absence , 2001, Proceedings 2001 Australian Software Engineering Conference.

[34]  Daniel Hoffman,et al.  ClassBench: A Methodology and Framework for Automated Class Testing , 1996 .

[35]  Tim Miller,et al.  Model-Based Specification Animation Using Testgraphs , 2002, ICFEM.