Improving test suites via operational abstraction

This paper presents the operational difference technique for generating, augmenting, and minimizing test suites. The technique is analogous to structural code coverage techniques, but it operates in the semantic domain of program properties rather than the syntactic domain of program text. The operational difference technique automatically selects test cases; it assumes only the existence of a source of test cases. The technique dynamically generates operational abstractions (which describe observed behavior and are syntactically identical to formal specifications)from test suite executions. Test suites can be generated by adding cases until the operational abstraction stops changing. The resulting test suites are as small, and detect as many faults, as suites with 100% branch coverage, and are better at detecting certain common faults. This paper also presents the area and stacking techniques for comparing test suite generation strategies; these techniques avoid bias due to test suite size.

[1]  Michael D. Ernst,et al.  Summary of dynamically discovering likely program invariants , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[2]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[3]  Michael D. Ernst,et al.  Invariant inference for static checking: , 2002, SIGSOFT '02/FSE-10.

[4]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[5]  Michael D. Ernst,et al.  Automatic generation of program specifications , 2002, ISSTA '02.

[6]  Michael R. Donat.  Automating Formal Specification-Based Testing , 1997, TAPSOFT.

[7]  A. Jefferson Offutt,et al.  An integrated automatic test data generation system , 1991, J. Syst. Integr..

[8]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[9]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[10]  Bev Littlewood,et al.  Evaluating Testing Methods by Delivered Reliability , 1998, IEEE Trans. Software Eng..

[11]  Thomas J. Ostrand,et al.  Automatic Generation of Test Scripts from Formal Test Specifications , 1989, Symposium on Testing, Analysis, and Verification.

[12]  Christophe Meudec,et al.  Automatic generation of software test cases from formal specifications , 1998 .

[13]  John B. Goodenough,et al.  Correction to "toward a theory of test data selection" , 1975, IEEE Transactions on Software Engineering.

[14]  Sriram Sankar,et al.  Specifying and Testing Software Components using ADL , 1994 .

[15]  Michael Harder,et al.  Improving Test Suites via Generated Specifications , 2002 .

[16]  Gregg Rothermel,et al.  An empirical study of regression test selection techniques , 2001, ACM Trans. Softw. Eng. Methodol..

[17]  RothermelGregg,et al.  Empirical Studies of a Safe Regression Test Selection Technique , 1998 .

[18]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[19]  RothermelGregg,et al.  An empirical study of regression test selection techniques , 2001 .

[20]  Michael D. Ernst,et al.  Invariant inference for static checking: an empirical evaluation , 2002, SOEN.

[21]  Simon Burton.  Towards Automated Unit Testing of Statechart Implementations , 1999 .

[22]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[23]  Daniel Hoffman,et al.  Boundary Values and Automated Component Testing , 1999, Softw. Test. Verification Reliab..

[24]  Daniel Hoffman,et al.  Tools and techniques for Java API testing , 2000, Proceedings 2000 Australian Software Engineering Conference.

[25]  R. Taylor,et al.  Partition testing does not inspire confidence , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[26]  Sriram Sankar,et al.  Structural specification-based testing with ADL , 1996, ISSTA '96.

[27]  Michael D. Ernst,et al.  Dynamically discovering likely program invariants , 2000 .

[28]  Jeffrey J. Joyce,et al.  GENERATING TEST CASES FROM FORMAL SPECIFICATIONS , 1996 .

[29]  Bev Littlewood,et al.  Correction to: Evaluating Testing Methods by Delivered Reliability , 1999 .

[30]  William G. Griswold,et al.  Quickly detecting relevant program invariants , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[31]  Nancy A. Lynch,et al.  Using simulated execution in verifying distributed algorithms , 2003, International Journal on Software Tools for Technology Transfer.

[32]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[33]  Mary Jean Harrold,et al.  Test-suite reduction and prioritization for modified condition/decision coverage , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[34]  Paul Ammann,et al.  A SPECIFICATION-BASED COVERAGE METRIC TO EVALUATE TEST SETS , 2001 .

[35]  Gregg Rothermel,et al.  Empirical Studies of a Safe Regression Test Selection Technique , 1998, IEEE Trans. Software Eng..

[36]  Debra J. Richardson,et al.  Approaches to specification-based testing , 1989 .

[37]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[38]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[39]  Richard G. Hamlet,et al.  Probable Correctness Theory , 1987, Inf. Process. Lett..

[40]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[41]  Shaoying Liu,et al.  Generating test data from SOFL specifications , 1999, J. Syst. Softw..

[42]  Gerard Salton,et al.  Automatic Information Organization And Retrieval , 1968 .

[43]  John B. Goodenough,et al.  Toward a theory of test data selection , 1975 .