Evaluation of the Prediction-Based Approach to Cost Reduction in Mutation Testing

Mutation testing is the most effective technique for assessing the quality of test suites, but it is also very expensive in terms of computational costs. The cost arises from the need to generate and execute a large number of so called mutants. The paper presents and evaluates a machine learning approach to dealing with the issue of limiting the number of executed mutants. The approach uses classification algorithm to predict mutants execution results for a subset of the generated mutants without their execution. The evaluation of the approach takes into consideration two aspects: accuracy of the predicted results and stability of prediction. In the paper the details of the evaluation experiment and its results are presented and discussed. The approach is tested on four examples having different number of mutants ranging from 90 to over 300. The obtained results indicate that the predicted value of the mutation score is consistently higher then the actual one thus allowing for using the results with high confidence.

[1]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[2]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[3]  Sang-Woon Kim,et al.  Mutation testing cost reduction by clustering overlapped mutants , 2016, J. Syst. Softw..

[4]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[5]  Joanna Strug,et al.  Using classification for cost reduction of applying mutation testing , 2017, 2017 Federated Conference on Computer Science and Information Systems (FedCSIS).

[6]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[7]  Joanna Strug,et al.  Machine Learning Approach in Mutation Testing , 2012, ICTSS.

[8]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[9]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[10]  Joanna Strug,et al.  Classifying Mutants with Decomposition Kernel , 2016, ICAISC.

[11]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[12]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[13]  Lu Zhang,et al.  Predictive Mutation Testing , 2016, IEEE Transactions on Software Engineering.

[14]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[15]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[16]  Mark Harman,et al.  Predictive Mutation Testing , 2019, IEEE Transactions on Software Engineering.

[17]  Macario Polo,et al.  Mutation Testing Cost Reduction Techniques: A Survey , 2010, IEEE Software.

[18]  Joanna Strug,et al.  Cost Reduction in Mutation Testing with Bytecode-Level Mutants Classification , 2018, ICAISC.

[19]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[20]  Yves Le Traon,et al.  An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).