LittleDarwin: A Feature-Rich and Extensible Mutation Testing Framework for Large and Complex Java Systems

Mutation testing is a well-studied method for increasing the quality of a test suite. We designed LittleDarwin as a mutation testing framework able to cope with large and complex Java software systems, while still being easily extensible with new experimental components. LittleDarwin addresses two existing problems in the domain of mutation testing: having a tool able to work within an industrial setting, and yet, be open to extension for cutting edge techniques provided by academia. LittleDarwin already offers higher-order mutation, null type mutants, mutant sampling, manual mutation, and mutant subsumption analysis. There is no tool today available with all these features that is able to work with typical industrial software systems.

[1]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[2]  John A. Clark,et al.  Class Mutation : Mutation Testing for Object-Oriented Programs , 2000 .

[3]  Anthony Ventresque,et al.  Demo: PIT a Practical Mutation Testing Tool for Java , 2016 .

[4]  Sarfraz Khurshid,et al.  Operator-based and random mutant selection: Better together , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[5]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[6]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[7]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[8]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[9]  Serge Demeyer,et al.  A Model to Estimate First-Order Mutation Coverage from Higher-Order Mutation Coverage , 2016, 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[10]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[11]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[13]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[14]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[15]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[16]  Lutz Prechelt,et al.  An Empirical Comparison of Seven Programming Languages , 2000, Computer.

[17]  Serge Demeyer,et al.  Evaluating random mutant selection at class-level in projects with non-adequate test suites , 2016, EASE.

[18]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[19]  Alex Groce,et al.  An empirical comparison of mutant selection approaches , 2015 .

[20]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[21]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[22]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[23]  Haidar Osman,et al.  Mining frequent bug-fix code changes , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[24]  John McGregor,et al.  Test Early, Test Often , 2007, J. Object Technol..

[25]  Serge Demeyer,et al.  Considering Polymorphism in Change-Based Test Suite Reduction , 2014, XP Workshops.

[26]  Patrick Joseph Walsh,et al.  A measure of test case completeness (software, engineering) , 1985 .

[27]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[28]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[29]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[30]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[31]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[32]  J. A. Acree.  On mutation , 1980 .

[33]  Bob Kurtz.  On the utility of dominator mutants for mutation testing , 2016, SIGSOFT FSE.

[34]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[35]  Yves Le Traon,et al.  Threats to the validity of mutation-based test assessment , 2016, ISSTA.

[36]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[37]  L. Darrell Whitley,et al.  HOMAJ: A Tool for Higher Order Mutation Testing in AspectJ and Java , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[38]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[39]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[40]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[41]  A. Jefferson Offutt,et al.  Inter-class mutation operators for Java , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[42]  Gilles Pokam,et al.  Selective mutation testing for concurrent code , 2013, ISSTA.

[43]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[44]  Ali Parsai,et al.  Mutation Analysis : An Industrial Experience Report , 2015 .