Mart: a mutant generation tool for LLVM

Program mutation makes small syntactic alterations to programs' code in order to artificially create faulty programs (mutants). Mutants creation (generation) tools are often characterized by their mutation operators and the way they create and represent the mutants. This paper presents Mart, a mutants generation tool, for LLVM bitcode, that supports the fine-grained definition of mutation operators (as matching rule - replacing pattern pair; uses 816 defined pairs by default) and the restriction of the code parts to mutate. New operators are implemented in Mart by implementing their matching rules and replacing patterns. Mart also implements in-memory Trivial Compiler Equivalence to eliminate equivalent and duplicate mutants during mutants generation. Mart generates mutant code as separated mutant files, meta-mutants file, weak mutation and mutant coverage instrumented files. Mart is publicly available (https://github.com/thierry-tct/mart). Mart has been applied to generate mutants for several research experiments and generated more than 4,000,000 mutants.

[1]  Abhik Roychoudhury,et al.  Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[2]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[3]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[4]  René Just,et al.  Inferring mutant utility from program context , 2017, ISSTA.

[5]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[6]  Goran Petrovic,et al.  State of Mutation Testing at Google , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[7]  Alex Denisov,et al.  Mull It Over: Mutation Testing Based on LLVM , 2018, 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[8]  Bo Wang,et al.  Faster mutation analysis via equivalence modulo states , 2017, ISSTA.

[9]  Abhik Roychoudhury,et al.  CoREBench: studying complexity of regression errors , 2014, ISSTA 2014.

[10]  Yves Le Traon,et al.  Assessing and Improving the Mutation Testing Practice of PIT , 2016, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[11]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[12]  Yves Le Traon,et al.  An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[13]  Koushik Sen,et al.  Selecting fault revealing mutants , 2018, Empirical Software Engineering.

[14]  Farah Hariri,et al.  SRCIROR: A Toolset for Mutation Testing of C Source Code and LLVM Intermediate Representation , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[15]  Yves Le Traon,et al.  Mutant Quality Indicators , 2018, 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).