Exploiting symmetries to test programs

Symmetries often appear as properties of many artifical settings. In program testing, they can be viewed as properties of programs and can be given by the tester to check the correctness of the computed outcome. In this paper, we consider symmetries to be permutation relations between program executions and use them to automate the testing process. We introduce a software testing paradigm called symmetric testing, where automatic test data generation is coupled with symmetries checking to uncover faults inside the programs. A practical procedure for checking that a program satisfies a given symmetry relation is described. The paradigm makes use of group theoretic results as a formal basis to minimize the number of outcome comparisons required by the method. This approach appears to be of particular interest for programs for which neither an oracle, nor any formal specification is available. We implemented symmetric testing by using the primitive operations of the Java unit testing tool Roast by N. Daley, D. Hoffman, and P. Strooper (2002). The experimental results we got on faulty versions of classical programs of the software testing community tend to show the effectiveness of the approach.

[1]  Premkumar T. Devanbu,et al.  Cryptographic Verification of Test Coverage Claims , 2000, IEEE Trans. Software Eng..

[2]  Jeffrey Voas,et al.  Certifying Off-the- Shelf Software , 1998 .

[3]  Marie-Claude Gaudel,et al.  Software testing based on formal specifications: a theory and a tool , 1991, Softw. Eng. J..

[4]  Tsong Yueh Chen,et al.  Fault-based testing in the absence of an oracle , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[5]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[6]  Jeffrey M. Voas,et al.  Certifying Off-the-Shelf Software Components , 1998, Computer.

[7]  Daniel Hoffman,et al.  A framework for table driven testing of Java classes , 2002, Softw. Pract. Exp..

[8]  Tsong Yueh Chen,et al.  Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing , 2002, ISSTA '02.

[9]  Karl N. Levitt,et al.  SELECT—a formal system for testing and debugging programs by symbolic execution , 1975 .

[10]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[11]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[12]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[13]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[14]  Manuel Blum,et al.  Designing programs that check their work , 1989, STOC '89.

[15]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[16]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[17]  Keith W. Miller,et al.  Local exhaustive testing: a software reliability tool , 1992, ACM-SE 30.

[18]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[19]  Philip Wadler,et al.  Theorems for free! , 1989, FPCA.

[20]  Paul Ammann,et al.  Data Diversity: An Approach to Software Fault Tolerance , 1988, IEEE Trans. Computers.

[21]  M. A. Armstrong.  Groups and symmetry , 1988 .

[22]  Arnaud Gotlieb,et al.  Automated metamorphic testing , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.