The Impact of Source Test Case Selection on the Effectiveness of Metamorphic Testing

Metamorphic Testing (MT) aims to alleviate the oracle problem. In MT, testers define metamorphic relations (MRs) which are used to generate new test cases (referred to as follow-up test cases) from the available test cases (referred to as source test cases). Both source and follow-up test cases are executed and their outputs are verified against the relevant MRs, of which any violation implies that the software under test is faulty. So far, the research on the effectiveness of MT has been focused on the selection of better MRs (that is, MRs that are more likely to be violated). In addition to MR selection, the source and follow-up test cases may also affect the effectiveness of MT. Since follow-up test cases are defined by the source test cases and MRs, selection of source test cases will then affect the effectiveness of MT. However, in existing MT studies, random testing is commonly adopted as the test case selection strategy for source test cases. This study aims to investigate the impact of source test cases on the effectiveness of MT. Since Adaptive Random Testing (ART) has been developed as an enhancement to Random Testing (RT), this study will focus on comparing the performance of RT and ART as source test case selection strategies on the effectiveness of MT. Experiment results show that ART outperforms RT on enhancing the effectiveness of MT.

[1]  Tsong Yueh Chen,et al.  On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions , 2013, 2013 13th International Conference on Quality Software.

[2]  Baowen Xu,et al.  Metamorphic slice: An application in spectrum-based fault localization , 2013, Inf. Softw. Technol..

[3]  Tsong Yueh Chen,et al.  Automated functional testing of online search services , 2012, Softw. Test. Verification Reliab..

[4]  Tsong Yueh Chen,et al.  Testing of Heuristic Methods: A Case Study of Greedy Algorithm , 2008, CEE-SET.

[5]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[6]  Dave Towey,et al.  Forgetting Test Cases , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[7]  Tsong Yueh Chen,et al.  On the statistical properties of testing effectiveness measures , 2006, J. Syst. Softw..

[8]  T. Y. Chen,et al.  Adaptive Random Testing , 2004, ASIAN.

[9]  Tsong Yueh Chen,et al.  Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing , 2002, ISSTA '02.

[10]  Tsong Yueh Chen,et al.  Proportional sampling strategy: a compendium and some insights , 2001, J. Syst. Softw..

[11]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[12]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[13]  Peter G. Bishop,et al.  The variation of software survival time for different operational input profiles (or why you can wait a long time for a big bug to fail) , 1993, FTCS-23 The Twenty-Third International Symposium on Fault-Tolerant Computing.

[14]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[15]  Paul Ammann,et al.  Data Diversity: An Approach to Software Fault Tolerance , 1988, IEEE Trans. Computers.

[16]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[17]  Tsong Yueh Chen,et al.  A Novel Linear-Order Algorithm for Adaptive Random Testing of Programs with Non-Numeric Inputs , 2014 .

[18]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[19]  Arlinta Christy Barus,et al.  An in-depth study of adaptive random testing for testing program with complex input types , 2010 .

[20]  Robert G. Merkel,et al.  Analysis and enhancements of adaptive random testing , 2005 .

[21]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .