A Model to Estimate First-Order Mutation Coverage from Higher-Order Mutation Coverage

The test suite is essential for fault detection during software development. First-order mutation coverage is an accurate metric to quantify the quality of the test suite. However, it is computationally expensive. Hence, the adoption of this metric is limited. In this study, we address this issue by proposing a realistic model able to estimate first-order mutation coverage using only higher-order mutation coverage. Our study shows how the estimation evolves along with the order of mutation. We validate the model with an empirical study based on 17 open-source projects.

[1]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[2]  Serge Demeyer,et al.  Mutation testing as a safety net for test code refactoring , 2015, XP Workshops.

[3]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[4]  Mike Papadakis,et al.  Evaluating Mutation Testing Alternatives: A Collateral Experiment , 2010, 2010 Asia Pacific Software Engineering Conference.

[5]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second‐order mutants , 2009, Softw. Test. Verification Reliab..

[6]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[7]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[8]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[9]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[10]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[11]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[12]  Vahid Garousi,et al.  A survey of software testing practices in Canada , 2013, J. Syst. Softw..

[13]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[14]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[15]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[16]  Mark Harman,et al.  Source code analysis and manipulation , 2002, Inf. Softw. Technol..

[17]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[18]  Serge Demeyer,et al.  Evaluating random mutant selection at class-level in projects with non-adequate test suites , 2016, EASE.

[19]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[20]  Ngoc Thanh Nguyen,et al.  Advanced Computational Methods for Knowledge Engineering , 2013, Advanced Computational Methods for Knowledge Engineering.

[21]  A Jeeerson Ooutt,et al.  Subsumption of Condition Coverage Techniques by Mutation Testing , 1996 .

[22]  John McGregor,et al.  Test Early, Test Often , 2007, J. Object Technol..

[23]  K. S. How Tai Wah.  A Theoretical Study of Fault Coupling , 2000, Softw. Test. Verification Reliab..

[24]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[25]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[26]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[27]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[28]  K. S. How Tai Wah,et al.  An analysis of the coupling effect I: single test data , 2003, Sci. Comput. Program..

[29]  B. Everitt.  The Cambridge Dictionary of Statistics , 1998 .

[30]  A. Jefferson Offutt.  The Coupling Effect: Fact or Fiction , 1989, Symposium on Testing, Analysis, and Verification.

[31]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..