Applying formal methods to software testing

This thesis examines applying formal methods to software testing. Software testing is a critical phase of the software life-cycle which can be very effective if performed rigorously. Formal specifications offer the bases for rigorous testing practices. Not surprisingly, the most immediate use of formal specifications in software testing is as sources of black-box test suites. However, formal specifications have more uses in software testing than merely being sources for test data. We examine these uses, and show how to get more assistance and benefit from formal methods in software testing. At the core of this work is a exible framework in which to conduct specification-based testing. The framework is founded on formal definitions of tests and test suites, which directly addresses important issues in managing software testing. This provides a uniform platform for other applications of formal methods to testing such as analysis and reification of tests, and also for applications beyond testing such as maintenance and specification validation. The framework has to be exible so that any testing strategies can be used. We examine the need to adapt certain strategies to work with the framework and formal specification. Our experiments showed some deficiencies that arise when using derivation strategies on abstract specifications. These deficiencies led us to develop two new specification-based testing strategies based on extensions to existing strate- gies. We demonstrate the framework, strategies, and other applications of formal methods to software testing using three case studies. In each of these, the framework was easy to use. It provided an elegant and powerful means for defining and structuring tests, and a suitable staging ground for other applications of formal methods to software testing. This thesis demonstrates how formal specification techniques can systematise the application of testing strategies, and also how the concepts of software testing can be combined with formal specifications to extend the role of the formal specification in software development.

[1]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[2]  Ian Hayes,et al.  Deriving Modular Designs from Formal Specifications Deriving Modular Designs from Formal Speciications , 1993 .

[3]  R. Taylor,et al.  Partition testing does not inspire confidence , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[4]  Jeffrey M. Voas,et al.  Faults on its sleeve: amplifying software reliability testing , 1993, ISSTA '93.

[5]  Jan K. Pachl,et al.  A notation for specifying test selection criteria , 1990, PSTV.

[6]  Mary Rees,et al.  A testing methodology framework , 1990, Proceedings., Fourteenth Annual International Computer Software and Applications Conference.

[7]  Stuart H. Zweben,et al.  Analysis and Refinement of Software Test Data Adequacy Properties , 1991, IEEE Trans. Software Eng..

[8]  Clazien D. Wezeman,et al.  Deriving Tests for Objects Specified in Z , 1992, Z User Workshop.

[9]  Richard G. Hamlet,et al.  Theoretical comparison of testing methods , 1989, TAV3.

[10]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[11]  David A. Carrington,et al.  Test templates: a specification-based testing framework , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[12]  Lori A. Clarke,et al.  Partition Analysis: A Method Combining Testing and Verification , 1985, IEEE Transactions on Software Engineering.

[13]  Pierre Dauchy,et al.  Test Data Selection From Algebraic Specifications: Application to an Automatic Subway Module , 1991, ESEC.

[14]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[15]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[16]  Cliff B. Jones,et al.  Systematic software development using VDM , 1986, Prentice Hall International Series in Computer Science.

[17]  Lori A. Clarke,et al.  A Close Look at Domain Testing , 1982, IEEE Transactions on Software Engineering.

[18]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[19]  Richard G. Hamlet,et al.  Probable Correctness Theory , 1987, Inf. Process. Lett..

[20]  William E. Howden,et al.  Functional program testing and analysis , 1986 .

[21]  John D. Gannon,et al.  Combining Testing with Formal Specifications: A Case Study , 1983, IEEE Transactions on Software Engineering.

[22]  William C. Hetzel,et al.  The complete guide to software testing , 1984 .

[23]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.

[24]  Luc Bougé,et al.  A Contribution to the Theory of Program Testing , 1985, Theor. Comput. Sci..

[25]  Roy S. Freedman,et al.  Testability of Software Components , 1991, IEEE Trans. Software Eng..

[26]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[27]  John S. Gourlay.  A Mathematical Framework for the Investigation of Testing , 1983, IEEE Transactions on Software Engineering.

[28]  Brad J. Cox,et al.  Planning the software industrial revolution , 1990, IEEE Software.

[29]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[30]  Richard G. Hamlet,et al.  Data Abstraction, Implementation, Specification, and Testing , 1981, TOPL.

[31]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[32]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[33]  David A. Carrington,et al.  A Tale of Two Paradigms: Formal Methods and Software Testing , 1994, Z User Workshop.

[34]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[35]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[36]  Andrew S. Tanenbaum,et al.  In defense of program testing or correctness proofs considered harmful , 1976, SIGP.

[37]  William E. Howden,et al.  A functional approach to program testing and analysis , 1986, IEEE Transactions on Software Engineering.

[38]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[39]  Richard A. Kemmerer,et al.  Testing Formal Specifications to Detect Design Errors , 1985, IEEE Transactions on Software Engineering.

[40]  Derek Coleman,et al.  Formal Specification and Design Time Testing , 1990, IEEE Trans. Software Eng..

[41]  Thomas J. Ostrand,et al.  Automatic Generation of Test Scripts from Formal Test Specifications , 1989, Symposium on Testing, Analysis, and Verification.

[42]  Debra J. Richardson,et al.  Approaches to specification-based testing , 1989 .

[43]  Elaine J. Weyuker,et al.  Theories of Program Testing and the Application of Revealing Subdomains , 1980, IEEE Transactions on Software Engineering.

[44]  Juha Kuusela,et al.  Supporting testing of specifications and implementations , 1990 .

[45]  David A. Carrington,et al.  Test template framework: a specification-based testing case study , 1993, ISSTA '93.

[46]  Laurent Fribourg,et al.  Test sets generation from algebraic specifications using logic programming , 1986, J. Syst. Softw..

[47]  Graeme Smith,et al.  The object-Z specification language: version 1 , 1991 .

[48]  James Lyle Peterson,et al.  Petri net theory and the modeling of systems , 1981 .

[49]  Lee J. White,et al.  A Domain Strategy for Computer Program Testing , 1980, IEEE Transactions on Software Engineering.

[50]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[51]  Paul Ammann,et al.  Using Z specifications in category partition testing , 1992, COMPASS `92 Proceedings of the Seventh Annual Conference on Computer Assurance.

[52]  Joseph A. Goguen,et al.  Parameterized Programming , 1984, IEEE Transactions on Software Engineering.

[53]  P.A.V. Hall,et al.  Towards testing with respect to formal specification , 1988 .

[54]  Edsger W. Dijkstra,et al.  Structured programming , 1972, A.P.I.C. Studies in data processing.

[55]  Ian J. Hayes,et al.  Specification directed module testing , 1986, IEEE Transactions on Software Engineering.

[56]  David Carrington,et al.  Deriving Software Test Cases from Formal Specifications , 1991 .

[57]  John E. Nicholls,et al.  Z Base Standard, Verson 1.0 , 1992 .

[58]  Elaine J. Weyuker,et al.  Some observations on partition testing , 1989, TAV3.