An Investigation of Compression Techniques to Speed up Mutation Testing

Mutation testing is widely considered as a high-end test coverage criterion due to the vast number of mutants it generates. Although many efforts have been made to reduce the computational cost of mutation testing, in practice, the scalability issue remains. In this paper, we explore whether we can use compression techniques to improve the efficiency of strong mutation based on weak mutation information. Our investigation is centred around six mutation compression strategies that we have devised. More specifically, we adopt overlapped grouping and Formal Concept Analysis (FCA) to cluster mutants and test cases based on the reachability (code coverage) and necessity (weak mutation) conditions. Moreover, we leverage mutation knowledge (mutation locations and mutation operator types) during compression. To evaluate our method, we conducted a study on 20 open source Java projects using manually written tests. We also compare our method with pure random sampling and weak mutation. The overall results show that mutant compression techniques are a better choice than random sampling and weak mutation in practice: they can effectively speed up strong mutation 6.3 to 94.3 times with an accuracy of >90%.

[1]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[2]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[3]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[4]  Annibale Panichella,et al.  Java Unit Testing Tool Competition - Fifth Round , 2017, 2017 IEEE/ACM 10th International Workshop on Search-Based Software Testing (SBST).

[5]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[6]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[7]  I. Comparison.  Faster Mutation Testing Inspired by Test Prioritization and Reduction , 2013 .

[8]  Sang-Woon Kim,et al.  Mutation testing cost reduction by clustering overlapped mutants , 2016, J. Syst. Softw..

[9]  Francisco Herrera,et al.  A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[10]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[11]  Urko Rueda,et al.  Java Unit Testing Tool Competition - Seventh Round , 2019, 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST).

[12]  Michael D. Ernst,et al.  Efficient mutation analysis by propagating and partitioning infected execution states , 2014, ISSTA 2014.

[13]  Joanna Strug,et al.  Machine Learning Approach in Mutation Testing , 2012, ICTSS.

[14]  Alice Richardson,et al.  Nonparametric Statistics for Non‐Statisticians: A Step‐by‐Step Approach by Gregory W. Corder, Dale I. Foreman , 2010 .

[15]  A. Jefferson Offutt,et al.  Analyzing the validity of selective mutation with dominator mutants , 2016, SIGSOFT FSE.

[16]  Alex Groce,et al.  Mutation Reduction Strategies Considered Harmful , 2017, IEEE Transactions on Reliability.

[17]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[18]  A. Jefferson Offutt,et al.  A mutation carol: Past, present and future , 2011, Inf. Softw. Technol..

[19]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..

[20]  Rudolf Wille,et al.  Formal Concept Analysis as Mathematical Theory of Concepts and Concept Hierarchies , 2005, Formal Concept Analysis.

[21]  René Just,et al.  Using Non-redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[22]  Roland H. Untch,et al.  Mutation-based software testing using program schemata , 1992, ACM Southeast Regional Conference.

[23]  Paolo Tonella,et al.  Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets , 2018, IEEE Transactions on Software Engineering.

[24]  Alex Groce,et al.  On The Limits of Mutation Reduction Strategies , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[25]  Gordon Fraser,et al.  Achieving scalable mutation-based generation of whole test suites , 2015, Empirical Software Engineering.

[26]  Lu Zhang,et al.  Predictive Mutation Testing , 2016, IEEE Transactions on Software Engineering.

[27]  GORDON FRASER,et al.  A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite , 2014, ACM Trans. Softw. Eng. Methodol..

[28]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[29]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[30]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[31]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[32]  Zhu.  Speeding-Up Mutation Testing via Data Compression and State Infection , 2017 .

[33]  J. Zobel,et al.  Mutation Testing for the New Century , 2001, The Springer International Series on Advances in Database Systems.

[34]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[35]  J. A. Acree.  On mutation , 1980 .

[36]  Vi Andrew Jefferson Offutt,et al.  Automatic test data generation , 1988 .

[37]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[38]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[39]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..