Identifying Useful Mutants to Test Time Properties

Real-time systems have to be verified and tested for timely behavior as well as functional behavior. Thus, time is an extra dimension that adds to the complexity of software testing. A timed automata model with a model-checker can be used to generate timed test traces. To properly test the timely behavior, the set of test traces should challenge the different time constraints in the model. This paper describes and adapts mutation operators that target such time constraints in timed automata models. Time mutation operators apply a delta to the time constraints to help testers design tests that exceed the time constraints. We suggest that the size of this delta determines how easy the mutant is to kill and that the optimal delta varies by the program, mutation operator, and the individual mutant. To avoid trivial and equivalent time mutants, the delta should be set individually for each mutant. We discuss mutant subsumption and define the problem of finding dominator mutants in this new domain. In this position paper, we outline an iterative tuning process where a statistical model-checker, UPPAAL SMC, is used to: (i) create a tuned set of dominator time mutants, and (ii) generate test traces that kill the mutants.

[1]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[2]  Kim G. Larsen,et al.  Time-Optimal Real-Time Test Case Generation Using Uppaal , 2003, FATES.

[3]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[4]  M. Trakhtenbrot,et al.  New Mutations for Evaluation of Specification and Implementation Levels of Adequacy in Testing of Statecharts Models , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[5]  Bernhard K. Aichernig,et al.  MoMut::UML Model-Based Mutation Testing for UML , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[6]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[7]  René Just,et al.  Inferring mutant utility from program context , 2017, ISSTA.

[8]  Márcio Eduardo Delamaro,et al.  Mutation analysis testing for finite state machines , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[9]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[10]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[11]  A. Jefferson Offutt,et al.  Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[12]  Ferhat Khendek,et al.  Timed Wp-Method: Testing Real-Time Systems , 2002, IEEE Trans. Software Eng..

[13]  Florian Lorber,et al.  Time for Mutants - Model-Based Mutation Testing with Timed Automata , 2013, TAP@STAF.

[14]  Joanna Strug,et al.  Machine Learning Approach in Mutation Testing , 2012, ICTSS.

[15]  Marius Mikucionis,et al.  Formal Methods and Testing , 2008 .

[16]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[17]  Kim G. Larsen,et al.  Uppaal SMC tutorial , 2015, International Journal on Software Tools for Technology Transfer.

[18]  A. Jefferson Offutt,et al.  Automated Testing of Timeliness : A Case Study , 2007, Second International Workshop on Automation of Software Test (AST '07).

[19]  Robert M. Hierons,et al.  Mutation testing from probabilistic and stochastic finite state machines , 2009, J. Syst. Softw..

[20]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[21]  A. Jefferson Offutt,et al.  Mutation-based testing criteria for timeliness , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[22]  A. Jefferson Offutt,et al.  Improving logic-based testing , 2013, J. Syst. Softw..

[23]  Ana R. Cavalli,et al.  Specification of Timed EFSM Fault Models in SDL , 2007, FORTE.

[24]  Daniel Sundmark,et al.  Mutating aspect-oriented models to test cross-cutting concerns , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[25]  Robert M. Hierons,et al.  Specification Mutation Analysis for Validating Timed Testing Approaches Based on Timed Automata , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference.

[26]  A. Jefferson Offutt,et al.  Analyzing the validity of selective mutation with dominator mutants , 2016, SIGSOFT FSE.

[27]  Robert M. Hierons,et al.  Testing Real-Time Embedded Systems using Timed Automata based approaches , 2013, J. Syst. Softw..

[28]  András Márki,et al.  On strong mutation and subsuming mutants , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[29]  W. Eric Wong,et al.  Mutation Testing Applied to Validate Specifications Based on Petri Nets , 1995, FORTE.

[30]  Wang Yi,et al.  Timed Automata: Semantics, Algorithms and Tools , 2003, Lectures on Concurrency and Petri Nets.

[31]  Daniel Sundmark,et al.  Using mutation to design tests for aspect-oriented models , 2017, Inf. Softw. Technol..

[32]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..