Automated metamorphic testing

Usual techniques for automatic test data generation are based on the assumption that a complete oracle will be available during the testing process. However, there are programs for which this assumption is unreasonable. Recently, Chen et al. (1998, 2001) proposed to overcome this obstacle by using known relations over the input data and their unknown expected outputs to seek a subclass of faults inside the program. In this paper, we introduce an automatic testing framework able to check these so-called metamorphic relations. The framework makes use of constraint logic programming techniques to find test data that violate a given metamorphic-relation. Circumstances where it can also prove that the program satisfies this relation are presented. The first experimental results we got with a prototype tool build on the top of the test data generator INKA, show that this methodology can be completely automated.

[1]  Peter J. Stuckey,et al.  Programming with Constraints: An Introduction , 1998 .

[2]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[3]  Joxan Jaffar,et al.  Constraint logic programming , 1987, POPL '87.

[4]  Bernard Botella,et al.  A CLP Framework for Computing Structural Test Data , 2000, Computational Logic.

[5]  Tsong Yueh Chen,et al.  Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing , 2002, ISSTA '02.

[6]  Felix T.S. Chan,et al.  Application of metamorphic testing in numerical analysis , 1998, ICSE 1998.

[7]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[8]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[9]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[10]  Mats Carlsson,et al.  An Open-Ended Finite Domain Constraint Solver , 1997, PLILP.

[11]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[12]  Mark N. Wegman,et al.  Efficiently computing static single assignment form and the control dependence graph , 1991, TOPL.

[13]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[14]  Yahia Lebbah,et al.  Solving Constraints over Floating-Point Numbers , 2001, CP.

[15]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[16]  Tsong Yueh Chen,et al.  Fault-based testing in the absence of an oracle , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.