Dreaming up Metamorphic Relations: Experiences from Three Fuzzer Tools

Metamorphic testing requires the availability of a suitable set of metamorphic relations (MRs) for the application domain of interest. A software testing practitioner interested in using metamorphic testing is thus blocked unless they can devise a suitable set of MRs. In this paper we offer some practical advice on sources of inspiration for MRs, based on our experience building three fuzzing tools based on metamorphic testing: MF++, which supports automated testing of C++11 libraries, C4, which tests concurrency support in C11 compilers, and spirv-fuzz, which aims to find bugs in compilers for the SPIR-V programming language (mainly used in computer graphics). The MRs we have devised have taken inspiration from three main sources: (1) careful study of specification documents related to the libraries and programming languages that these tools target, (2) consultation of prior work and discussion with domain experts, and (3) manual inspection of the results of automated code coverage analysis on the systems under test. We describe these sources of inspiration in detail, giving a range of concrete examples for each. We hope that this experience report will help to inform developers of future metamorphic testing tools as to the steps they can take to discover MRs in their domains of interest.

[1]  Yingfei Xiong,et al.  A Survey of Compiler Testing , 2019 .

[2]  Xuan Liu,et al.  A New Method for Constructing Metamorphic Relations , 2012, 2012 12th International Conference on Quality Software.

[3]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[4]  Alessandra Gorla,et al.  Cross-checking oracles from intrinsic software redundancy , 2014, ICSE.

[5]  Alastair F. Donaldson,et al.  Many-core compiler fuzzing , 2015, PLDI.

[6]  Alessandra Gorla,et al.  Search-based synthesis of equivalent method sequences , 2014, SIGSOFT FSE.

[7]  Tao Guo,et al.  Security assurance with program path analysis and metamorphic testing , 2013, 2013 IEEE 4th International Conference on Software Engineering and Service Science.

[8]  Alastair F. Donaldson,et al.  Putting Randomized Compiler Testing into Production (Experience Report) , 2020, ECOOP.

[9]  Alastair F. Donaldson,et al.  Automated testing of graphics shader compilers , 2017, Proc. ACM Program. Lang..

[10]  Armin Biere,et al.  BTOR: bit-precise modelling of word-level problems for model checking , 2008, SMT '08/BPR '08.

[11]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[12]  Lu Zhang,et al.  Search-based inference of polynomial metamorphic relations , 2014, ASE.

[13]  Mary Lou Soffa,et al.  Automated test data generation using an iterative relaxation method , 1998, SIGSOFT '98/FSE-6.

[14]  Dave Towey,et al.  Metamorphic Relations for Enhancing System Understanding and Use , 2020, IEEE Transactions on Software Engineering.

[15]  Alastair F. Donaldson,et al.  Metamorphic Testing for (Graphics) Compilers , 2016, 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET).

[16]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[17]  Francesco Zappa Nardelli,et al.  Compiler testing via a theory of sound optimisations in the C11/C++11 memory model , 2013, PLDI.

[18]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[19]  Wei Wu,et al.  An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique , 2010, 2010 Asia Pacific Software Engineering Conference.

[20]  Bruno Dutertre,et al.  Yices 2.2 , 2014, CAV.

[21]  Sven Verdoolaege.  Integer Set Coalescing , 2015 .

[22]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[23]  Sven Verdoolaege,et al.  isl: An Integer Set Library for the Polyhedral Model , 2010, ICMS.

[24]  James M. Bieman,et al.  Using machine learning techniques to detect metamorphic relations for programs without test oracles , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).