Advantages of Mutation in Passive Testing: An Empirical Study

This paper presents an empirical study of the mutation techniques used by the tool PASTE. This tool allows the automation of our passive testing methodology for systems that present stochastic-time information. In our proposal, invariants express the fact that each time the implementation under test performs a given sequence of actions, then it must exhibit a behavior according to the probability distribution functions reflected in the invariant. We briefly review the theoretical framework of our methodology and the main features of our tool. Next, we present in detail the Mutants module that provides us with a functionality to test the effectiveness of invariants for detecting errors. Finally, we present a study of the results obtained from the performed experiments.

[1]  Mercedes G. Merayo,et al.  Passive Testing of Timed Systems , 2008, ATVA.

[2]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999 .

[3]  Ana R. Cavalli,et al.  A passive testing approach based on invariants: application to the WAP , 2005, Comput. Networks.

[4]  I. Lee,et al.  Automatic generation of tests for timing constraints from requirements , 1997, Proceedings Third International Workshop on Object-Oriented Real-Time Dependable Systems.

[5]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[6]  Paulo César Masiero,et al.  Mutation testing applied to validate specifications based on statecharts , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[7]  Ana R. Cavalli,et al.  Fast Testing of Critical Properties through Passive Testing , 2003, TestCom.

[8]  W. Eric Wong,et al.  Mutation Testing Applied to Validate SDL Specifications , 2004, TestCom.

[9]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[10]  Mercedes G. Merayo,et al.  Extending EFSMs to Specify and Test Timed Systems with Action Durations and Time-Outs , 2008, IEEE Transactions on Computers.

[11]  R. Hierons,et al.  Mutation Testing from Probabilistic Finite State Machines , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[12]  David A. Carrington,et al.  A Tale of Two Paradigms: Formal Methods and Software Testing , 1994, Z User Workshop.

[13]  Frits W. Vaandrager,et al.  Testing timed automata , 1997, Theor. Comput. Sci..

[14]  M. Ümit Uyar,et al.  Finite State Machine Based Formal Methods in Protocol Conformance Testing: From Theory to Implementation , 1991, Comput. Networks ISDN Syst..

[15]  Ferhat Khendek,et al.  Timed Wp-Method: Testing Real-Time Systems , 2002, IEEE Trans. Software Eng..

[16]  Jan Tretmans,et al.  Testing Concurrent Systems: A Formal Approach , 1999, CONCUR.

[17]  Mercedes G. Merayo,et al.  Extending EFSMs to Specify and Test Timed Systems with Action Durations and Timeouts , 2006, FORTE.

[18]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[19]  Ana R. Cavalli,et al.  New approaches for passive testing using an Extended Finite State Machine specification , 2003, Inf. Softw. Technol..

[20]  Richard Lai,et al.  A survey of communication protocol testing , 2002, J. Syst. Softw..

[21]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999, Softw. Test. Verification Reliab..

[22]  Mercedes G. Merayo,et al.  HOTL: Hypotheses and observations testing logic , 2007, J. Log. Algebraic Methods Program..

[23]  Manuel Núñez,et al.  Conformance Testing Relations for Timed Systems , 2005, FATES.

[24]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[25]  Mercedes G. Merayo,et al.  Passive Testing of Stochastic Timed Systems , 2009, 2009 International Conference on Software Testing Verification and Validation.

[26]  David Lee,et al.  Network protocol system monitoring-a formal approach with passive testing , 2006, IEEE/ACM Transactions on Networking.

[27]  A. J. Offutt.  A practical system for mutation testing: help for the common programmer , 1994, Proceedings., International Test Conference.

[28]  Sandro Morasca,et al.  Generating test cases for real-time systems from logic specifications , 1995, TOCS.

[29]  Teruo Higashino,et al.  Generating Test Cases for a Timed I/O Automaton Model , 1999, IWTCS.