SiMut: Exploring Program Similarity to Support the Cost Reduction of Mutation Testing

Scientists have created many cost reduction techniques for mutation testing, and most of them reduce cost with minor losses of effectiveness. However, many of these techniques are difficult to generalize, difficult to scale, or both. Published results are usually limited to a modest collection of programs. Therefore, an open question is whether the results of a given cost reduction technique on programs studied in the paper will hold true for other programs. This paper introduces a conceptual framework, named SiMut, to support the cost reduction of mutation testing based on historical data and program similarity. Given a new, untested program u, the central idea is applying to u the same cost reduction strategy applied to a group G of programs that are similar to u and have already been tested with mutation, and check for consistency of results in terms of reduced costs and quality of test sets. SiMut includes activities to compute program abstractions and similarity. Based on this information, it supports the application of mutation cost reduction techniques to both G and u. This paper presents the concepts behind SiMut, a proof-of-concept implementation of SiMut, and results from a pilot study. Finally, we discuss some issues related to the use of SiMut, focusing on the composition of a representative dataset to properly explore the potential of our framework.

[1]  Auri Marcelo Rizzo Vincenzi,et al.  A coevolutionary algorithm to automatic test case selection and mutant in Mutation Testing , 2013, 2013 IEEE Congress on Evolutionary Computation.

[2]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[3]  Nicos Malevris,et al.  Identifying More Equivalent Mutants via Code Similarity , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[4]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[5]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[6]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[7]  Andrew W. Moore,et al.  X-means: Extending K-means with Efficient Estimation of the Number of Clusters , 2000, ICML.

[8]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.

[9]  A. Jefferson Offutt,et al.  The class-level mutants of MuJava , 2006, AST '06.

[10]  Mark Harman,et al.  Detecting Trivial Mutant Equivalences via Compiler Optimisations , 2018, IEEE Transactions on Software Engineering.

[11]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[12]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[13]  Yves Le Traon,et al.  Chapter Six - Mutation Testing Advances: An Analysis and Survey , 2019, Adv. Comput..

[14]  Fabiano Cutigi Ferrari,et al.  An Evaluation of Internal Program Metrics as Predictors of Mutation Operator Score , 2019, SAST 2019.

[15]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[16]  Ian H. Witten,et al.  Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques , 2016 .

[17]  Koushik Sen,et al.  Selecting fault revealing mutants , 2018, Empirical Software Engineering.

[18]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[19]  Chanchal Kumar Roy,et al.  Comparison and evaluation of code clone detection techniques and tools: A qualitative approach , 2009, Sci. Comput. Program..

[20]  Yves Le Traon,et al.  How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults , 2017, Empirical Software Engineering.

[21]  Roland H. Untch.  On reduced neighborhood mutation analysis using a single mutagenic operator , 2009, ACM-SE 47.

[22]  A. Jefferson Offutt,et al.  Analyzing the validity of selective mutation with dominator mutants , 2016, SIGSOFT FSE.

[23]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[24]  Mark Harman,et al.  Angels and monsters: an empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation , 2014, ASE.

[25]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[26]  A. Jefferson Offutt,et al.  Empirical Evaluation of the Statement Deletion Mutation Operator , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[27]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[28]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[29]  Fabiano Cutigi Ferrari,et al.  A systematic literature review of techniques and metrics to reduce the cost of mutation testing , 2019, J. Syst. Softw..

[30]  Marcelo Medeiros Eler,et al.  Using a cluster analysis method for grouping classes according to their inferred testability: An investigation of CK metrics, code coverage and mutation score , 2017, 2017 36th International Conference of the Chilean Computer Science Society (SCCC).

[31]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[32]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[33]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[34]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[35]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[36]  Yves Le Traon,et al.  Predicting the fault revelation utility of mutants , 2018, ICSE.

[37]  Yves Le Traon,et al.  An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[38]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[39]  Mohammad El-Ramly,et al.  Similarity in Programs , 2006, Duplication, Redundancy, and Similarity in Software.

[40]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[41]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[42]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[43]  Akbar Siami Namin,et al.  The use of mutation in testing experiments and its sensitivity to external threats , 2011, ISSTA '11.

[44]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..