Mutomvo: Mutation testing framework for simulated cloud and HPC environments

Abstract Many current applications provide high performance to process large volumes of data. These applications usually run in highly distributed environments, like cloud and HPC systems. Nevertheless, the large and complex architectures required for deploying these applications may not be available during the development phase. This limitation can be overcome by using simulation platforms to model a wide range of distributed system configurations and execute these applications in the modeled system. Usually, these applications are tested against a small number of test cases that are manually designed by the testers. It is desirable to have effective test suites in order to detect failures in the application models. In this paper we propose a mutation testing framework for detecting errors in distributed applications executed in simulated environments. The execution of a test suite against the set of mutated models allows to determine its effectiveness for detecting different errors. The proposal has been implemented in a tool called MuTomVo. In order to support the feasibility of the proposal, we have carried out a case study over three applications running in different distributed systems: a client/server model, intensive computation and scientific pipeline.

[1]  Jie Li,et al.  eScience in the cloud: A MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[2]  Yves Le Traon,et al.  Threats to the validity of mutation-based test assessment , 2016, ISSTA.

[3]  Mahmoud Al-Ayyoub,et al.  Evaluating map reduce tasks scheduling algorithms over cloud computing infrastructure , 2015, Concurr. Comput. Pract. Exp..

[4]  Alexander L. Wolf,et al.  Simulation-Based Testing of Distributed Systems , 2006 .

[5]  Jesús Carretero,et al.  iCanCloud: A Brief Architecture Overview , 2012, 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications.

[6]  K. N. King,et al.  A Fortran 77 interpreter for mutation analysis , 1987, PLDI 1987.

[7]  Mercedes G. Merayo,et al.  A formal framework to analyze cost and performance in Map-Reduce based applications , 2014, J. Comput. Sci..

[8]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[9]  A. Varga,et al.  THE OMNET++ DISCRETE EVENT SIMULATION SYSTEM , 2003 .

[10]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[11]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[12]  Mercedes G. Merayo,et al.  EMINENT: EMbarrassINgly parallEl mutatioN Testing , 2016, ICCS.

[13]  A. Jefferson Offutt,et al.  Mutation operators for testing Android apps , 2017, Inf. Softw. Technol..

[14]  Chao Wang,et al.  CCmutator: A mutation generator for concurrency constructs in multithreaded C/C++ applications , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[15]  Alberto Sillitti,et al.  Using the Eclipse C/C++ Development Tooling as a Robust, Fully Functional, Actively Maintained, Open Source C++ Parser , 2012, OSS.

[16]  Inmaculada Medina-Bulo,et al.  Class mutation operators for C++ object-oriented systems , 2015, Ann. des Télécommunications.

[17]  Fan Wu,et al.  Mutation testing of memory-related operators , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[18]  Robert M. Hierons,et al.  A methodology for validating cloud models using metamorphic testing , 2015, Ann. des Télécommunications.

[19]  Mercedes G. Merayo,et al.  Passive testing of communicating systems with timeouts , 2015, Inf. Softw. Technol..

[20]  Jesús Carretero,et al.  SIMCAN: A flexible, scalable and expandable simulation platform for modelling and simulating distributed architectures and applications , 2012, Simul. Model. Pract. Theory.

[21]  Yong Rae Kwon,et al.  MuJava: an automated class mutation system: Research Articles , 2005 .

[22]  D. Gros,et al.  The Global Economy in 2030: Trends and Strategies for Europe , 2014 .

[23]  John Shalf,et al.  Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[24]  Fan Wu,et al.  Memory mutation testing , 2017, Inf. Softw. Technol..

[25]  Aditya P. Mathur,et al.  Interface Mutation: An Approach for Integration Testing , 2001, IEEE Trans. Software Eng..

[26]  Bertrand Meyer,et al.  On the number and nature of faults found by random testing , 2011, Softw. Test. Verification Reliab..

[27]  Inmaculada Medina-Bulo,et al.  Assessment of C++ object‐oriented mutation operators: A selective mutation approach , 2017, Softw. Test. Verification Reliab..

[28]  Sang-Woon Kim,et al.  Mutation testing cost reduction by clustering overlapped mutants , 2016, J. Syst. Softw..

[29]  Lionel C. Briand,et al.  Random Testing: Theoretical Results and Practical Implications , 2012, IEEE Transactions on Software Engineering.

[30]  Juan de Lara,et al.  OUTRIDER: Optimizing the mUtation Testing pRocess In Distributed EnviRonments , 2017, ICCS.

[31]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[32]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.