Mutation at the multi-class and system levels

Mutation analysis has been applied to many testing problems, including numerous programming languages, specifications, network protocols, web services, and security policies. Program mutation, where mutation analysis is applied to programs, has been applied to the unit level (functions and methods), integration of pairs of functions, and individual classes. However, program mutation has not been applied to the problem of integration testing of multiple classes or entire software programs; thus, there is no system level mutation. This paper introduces a project on the problem of integration testing of multiple classes (multi-class) and system level mutation testing. The technical differences between using mutation to test single classes and multiple classes are explored, and new system level mutation operators are defined. A new execution style for detecting killed mutants, flexible weak mutation, is introduced. A mutation tool, Bacterio, still under construction, is also described.

[1]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[2]  M. R. Woodward,et al.  From weak to strong, dead or alive? an analysis of some mutation testing issues , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[3]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[4]  José Carlos Maldonado,et al.  Proteum/IM 2.0: An Integrated Mutation Testing Environment , 2001 .

[5]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[6]  Yu-Seung Ma.  Description of Method-level Mutation Operators for Java , 2011 .

[7]  A. Jefferson Offutt,et al.  An Analysis of OO Mutation Operators , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[8]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[9]  J. Offutt,et al.  Mutation testing implements grammar-based testing , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[10]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999 .

[11]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[12]  Javier Tuya,et al.  Mutating database queries , 2007, Inf. Softw. Technol..

[13]  Martin R. Woodward,et al.  An integrated system for program testing using weak mutation and data flow analysis , 1985, ICSE '85.

[14]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[15]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[16]  Aditya P. Mathur,et al.  Interface Mutation: An Approach for Integration Testing , 2001, IEEE Trans. Software Eng..

[17]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[18]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[19]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[20]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[21]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[22]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[23]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[24]  Anna Derezinska.  Advanced mutation operators applicable in C# programs , 2006, SET.

[25]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[26]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[27]  S. Inglis,et al.  Jumble Java Byte Code to Measure the Effectiveness of Unit Tests , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[28]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[29]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[30]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[31]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[32]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[33]  A. J. Offutt.  A practical system for mutation testing: help for the common programmer , 1994, Proceedings., International Test Conference.

[34]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[35]  Brian Marick,et al.  The weak mutation hypothesis , 1991, TAV4.

[36]  Teemu Kanstrén.  Towards a deeper understanding of test coverage , 2008, J. Softw. Maintenance Res. Pract..

[37]  Macario Polo,et al.  Mutation Testing Cost Reduction Techniques: A Survey , 2010, IEEE Software.

[38]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..