How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?

In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, properties of the software under test represented in the form of relations among inputs and outputs of multiple executions, to help verify the correctness of a program. This paper presents new empirical evidence to support this approach, which has been used to alleviate the oracle problem in various applications and to enhance several software analysis and testing techniques. It has been observed that identification of a sufficient number of appropriate metamorphic relations for testing, even by inexperienced testers, was possible with a very small amount of training. Furthermore, the cost-effectiveness of the approach could be enhanced through the use of more diverse metamorphic relations. The empirical studies presented in this paper clearly show that a small number of diverse metamorphic relations, even those identified in an ad hoc manner, had a similar fault-detection capability to a test oracle, and could thus effectively help alleviate the oracle problem.

[1]  Barton P. Miller,et al.  An empirical study of the robustness of MacOS applications using random testing , 2006, RT '06.

[2]  Nancy G. Leveson,et al.  An experimental evaluation of the assumption of independence in multiversion programming , 1986, IEEE Transactions on Software Engineering.

[3]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[4]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[5]  Tsong Yueh Chen,et al.  Testing of Heuristic Methods: A Case Study of Greedy Algorithm , 2008, CEE-SET.

[6]  David Lorge Parnas,et al.  A Comparison of Tabular Expression-Based Testing Strategies , 2011, IEEE Transactions on Software Engineering.

[7]  Gregory Gay,et al.  Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[8]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .

[9]  Wei Wu,et al.  An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique , 2010, 2010 Asia Pacific Software Engineering Conference.

[10]  Tsong Yueh Chen,et al.  Testing embedded software by metamorphic testing: A wireless metering system case study , 2011, 2011 IEEE 36th Conference on Local Computer Networks.

[11]  Huai Liu,et al.  Metamorphic Testing for Web Services: Framework and a Case Study , 2011, 2011 IEEE International Conference on Web Services.

[12]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[13]  Arnaud Gotlieb,et al.  Automated metamorphic testing , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[14]  Brent Hailpern,et al.  Software debugging, testing, and verification , 2002, IBM Syst. J..

[15]  Huai Liu,et al.  Conformance Testing of Network Simulators Based on Metamorphic Testing Technique , 2009, FMOODS/FORTE.

[16]  S. Reid.  The Art of Software Testing, Second edition. Glenford J. Myers. Revised and updated by Tom Badgett and Todd M. Thomas, with Corey Sandler. John Wiley and Sons, New Jersey, U.S.A., 2004. ISBN: 0-471-46912-2, pp 234: Book Reviews , 2005 .

[17]  Gail E. Kaiser,et al.  Automatic system testing of programs without test oracles , 2009, ISSTA.

[18]  Derrick G. Kourie,et al.  Software testing using model programs , 2001, Softw. Pract. Exp..

[19]  MaYu-Seung,et al.  MuJava: an automated class mutation system , 2005 .

[20]  Paul Ammann,et al.  Data Diversity: An Approach to Software Fault Tolerance , 1988, IEEE Trans. Computers.

[21]  Algirdas Avizienis,et al.  The N-Version Approach to Fault-Tolerant Software , 1985, IEEE Transactions on Software Engineering.

[22]  Sergio Segura,et al.  Automated metamorphic testing on the analyses of feature models , 2011, Inf. Softw. Technol..

[23]  Baowen Xu,et al.  Metamorphic slice: An application in spectrum-based fault localization , 2013, Inf. Softw. Technol..

[24]  Robert M. Hierons.  Oracles for Distributed Testing , 2012, IEEE Transactions on Software Engineering.

[25]  T. H. Tse,et al.  An empirical comparison between direct and indirect test result checking approaches , 2006, SOQUA '06.

[26]  Tsong Yueh Chen,et al.  Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging , 2011, IEEE Transactions on Software Engineering.

[27]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[28]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[29]  David S. Rosenblum.  A Practical Approach to Programming With Assertions , 1995, IEEE Trans. Software Eng..

[30]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[31]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[32]  Lionel C. Briand,et al.  Achieving scalable model-based testing through test case diversity , 2013, TSEM.

[33]  W. K. Chan,et al.  Experimental study to compare the use of metamorphic testing and assertion checking , 2009 .

[34]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[35]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[36]  Johannes Mayer,et al.  An Empirical Study on the Selection of Good Metamorphic Relations , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[37]  Kai-Yuan Cai,et al.  Impacts of Test Suite's Class Imbalance on Spectrum-Based Fault Localization Techniques , 2013, 2013 13th International Conference on Quality Software.

[38]  Sami Beydeda.  Self-Metamorphic-Testing Components , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[39]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[40]  W. Press,et al.  Numerical Recipes in Fortran: The Art of Scientific Computing.@@@Numerical Recipes in C: The Art of Scientific Computing. , 1994 .

[41]  Tsong Yueh Chen,et al.  On the Relationship Between Partition and Random Testing , 1994, IEEE Trans. Software Eng..