Supporting the software testing process through specification animation

Achieving consistency between a specification and its implementation is an important part of software development. In this paper, we present a method for generating passive test oracles that act as self-checking implementations. The implementation is verified using an animation tool to check that the behavior of the implementation matches the behavior of the specification. We discuss how to integrate this method into a framework developed for systematically animating specifications, which means a tester can significantly reduce testing time and effort by reusing work products from the animation. One such work product is a testgraph: a directed graph that partially models the states and transitions of the specification. Testgraphs are used to generate sequences for animation, and during testing, to execute these same sequences on the implementation.

[1]  Colin O'Halloran,et al.  Experiences with PiZA, an Animator for Z , 1997, ZUM.

[2]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[3]  Roger Duke,et al.  Formal Object Oriented Specification Using Object-Z , 2000 .

[4]  Leesa Murray,et al.  From Object-Z Specifications to ClassBench Test Suites , 2000, Softw. Test. Verification Reliab..

[5]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[6]  Tim Miller,et al.  A framework and tool support for the systematic testing of model-based specifications , 2003, TSEM.

[7]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[8]  Paul A. Strooper,et al.  Requirements engineering and verification using specification animation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[9]  Hans-Martin Hörcher,et al.  Improving Software Tests Using Z Specifications , 1995, ZUM.

[10]  Susan Stepney.  Testing as Abstraction , 1995, ZUM.

[11]  David J. Robson,et al.  A State-Based Approach to the Testing of Class-Based Programs , 1995, Softw. Concepts Tools.

[12]  S. Easterbrook,et al.  Generating Test Oracles via Model Checking , 1997 .

[13]  Paul Strooper,et al.  From Object‐Z specifications to ClassBench test suites , 2000 .

[14]  Tim Miller,et al.  Animation can show only the presence of errors, never their absence , 2001, Proceedings 2001 Australian Software Engineering Conference.

[15]  Paul A. Strooper,et al.  Possum: an animator for the SUM specification language , 1997, Proceedings of Joint 4th International Computer Science Conference and 4th Asia Pacific Software Engineering Conference.

[16]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[17]  Jason McDonald,et al.  Translating Object-Z specifications to passive test oracles , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[18]  P. Strooper,et al.  Unit Operations for Automated Class TestingNigel , 2000 .

[19]  David A. Carrington,et al.  A Framework for Specification-Based Testing , 1996, IEEE Trans. Software Eng..

[20]  Hélène Waeselynck,et al.  B model animation for external verification , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[21]  Daniel Hoffman,et al.  ClassBench: A Methodology and Framework for Automated Class Testing , 1996 .

[22]  Bernhard K. Aichernig,et al.  Formal specification techniques as a catalyst in validation , 2000, Proceedings. Fifth IEEE International Symposium on High Assurance Systems Engineering (HASE 2000).

[23]  Tim Miller,et al.  Model-Based Specification Animation Using Testgraphs , 2002, ICFEM.

[24]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[25]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[26]  Phyllis G. Frankl,et al.  The ASTOOT approach to testing object-oriented programs , 1994, TSEM.

[27]  Fabrice Bouquet,et al.  BZ-TT: A tool-set for test generations from Z and B using constraint logic programming , 2002 .

[28]  Robert M. Hierons.  Testing from a Z Specification , 1997, Softw. Test. Verification Reliab..

[29]  Michael Winikoff,et al.  Verifying model oriented specifications through animation , 1998, Proceedings 1998 Asia Pacific Software Engineering Conference (Cat. No.98EX240).

[30]  Tim Miller,et al.  Combining the animation and testing of abstract data types , 2001, Proceedings Second Asia-Pacific Conference on Quality Software.