Is Mutation Testing Ready to Be Adopted Industry-Wide?

Mutation Testing has a long research history as a way to improve the quality of software tests. However, it has not yet reached wide consensus for industry-wide adoption, mainly due to missing clear benefits and computational complexity for the application to large systems. In this paper, we investigate the current state of mutation testing support for Java Virtual Machine (JVM) environments. By running an experimental evaluation, we found out that while default configurations are unbearable for larger projects, using strategies such as selective operators, second order mutation and multi-threading can increase the applicability of the approach. However, there is a trade-off in terms of quality of the achieved results of the mutation analysis process that needs to be taken into account.

[1]  Vaclav Dedik,et al.  Automated Bug Triaging in an Industrial Context , 2016, 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[2]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[3]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[4]  Franz Wotawa,et al.  Is Mutation Testing Scalable for Real-World Software Projects? , 2011 .

[5]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[6]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[7]  Sunil Kumar Khatri,et al.  Experimental comparison of automated mutation testing tools for java , 2015, 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions).

[8]  Lech Madeyski,et al.  Judy - a mutation testing tool for Java , 2010, IET Softw..

[9]  Bruno Rossi,et al.  Towards an Improvement of Bug Severity Classification , 2014, 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications.

[10]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[11]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[12]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[13]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[14]  Mickaël Delahaye,et al.  A Comparison of Mutation Analysis Tools for Java , 2013, 2013 13th International Conference on Quality Software.

[15]  Lisa Crispin,et al.  Agile Testing: A Practical Guide for Testers and Agile Teams , 2008 .