MD-ART: a test case generation method without test oracle problem

Adaptive random testing (ART), as an improved random testing method, preserves the advantages of traditional random test method and overcomes the blindness of traditional random testing method. But it is usually not easy to validate the correctness of the output, except for some special test cases. In other words, the test oracle problem is unresolved. In this research, we introduced metamorphic testing (MT) and metamorphic distance into ART, which is called metamorphic distance based ART (MD-ART) to provide the test oracle. The results of primary experiment results show that MD-ART performs better than traditional MT and ART not only in test effectiveness but also in test efficiency and test coverage.

[1]  Gagandeep,et al.  An Automated Metamorphic Testing Technique for Designing Effective Metamorphic Relations , 2012, IC3.

[2]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[3]  Dave Towey,et al.  A revisit of adaptive random testing by restriction , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[4]  Johannes Mayer,et al.  An Empirical Study on the Selection of Good Metamorphic Relations , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[5]  Tsong Yueh Chen,et al.  Testing a binary space partitioning algorithm with metamorphic testing , 2011, SAC '11.

[6]  Shin Yoo.  Metamorphic Testing of Stochastic Optimisation , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[7]  W. K. Chan,et al.  Experimental study to compare the use of metamorphic testing and assertion checking , 2009 .

[8]  Tsong Yueh Chen,et al.  Adaptive random testing through dynamic partitioning , 2004, Fourth International Conference onQuality Software, 2004. QSIC 2004. Proceedings..

[9]  Lizhi Cai,et al.  An optimized method for generating cases of metamorphic testing , 2012, 2012 6th International Conference on New Trends in Information Science, Service Science and Data Mining (ISSDM2012).

[10]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[11]  Tsong Yueh Chen,et al.  On the statistical properties of testing effectiveness measures , 2006, J. Syst. Softw..

[12]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[13]  Tsong Yueh Chen,et al.  Proportional sampling strategy: guidelines for software testing practitioners , 1996, Inf. Softw. Technol..

[14]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[15]  Peng Wu.  Metamorphic Testing and Special Case Testing: A Case Study , 2005 .

[16]  Baowen Xu,et al.  An Effective Iterative Metamorphic Testing Algorithm Based on Program Path Analysis , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[17]  Song Huang,et al.  A Formal Model for Metamorphic Relation Decomposition , 2013, 2013 Fourth World Congress on Software Engineering.

[18]  Zuohua Ding,et al.  Testing Central Processing Unit scheduling algorithms using Metamorphic Testing , 2013, 2013 IEEE 4th International Conference on Software Engineering and Service Science.

[19]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[20]  Lu Zhang,et al.  Search-based inference of polynomial metamorphic relations , 2014, ASE.

[21]  Ying Liu,et al.  Metamorphic Testing and Testing with Special Values , 2004, SNPD.

[22]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[23]  Sandeep Kang,et al.  Metamorphic Testing : Using the Properties of Sut , 2011 .

[24]  Peng Wu,et al.  Iterative Metamorphic Testing , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[25]  Robert G. Merkel,et al.  Analysis and enhancements of adaptive random testing , 2005 .

[26]  Tsong Yueh Chen,et al.  On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions , 2013, 2013 13th International Conference on Quality Software.

[27]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[28]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[29]  Upulee Kanewala.  Techniques for Automatic Detection of Metamorphic Relations , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[30]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[31]  Tsong Yueh Chen,et al.  An upper bound on software testing effectiveness , 2008, TSEM.