Coverage-Based Testing with Symbolic Transition Systems

We provide a model-based testing approach for systems comprising both state-transition based control flow, and data elements such as variables and data-dependent transitions. We propose test generation and execution, based on model-coverage: we generate test cases that aim to reach all transitions of the model. To obtain a test case reaching a certain transition, we need to combine reachability in the control flow, and satisfiability of the data elements of the model. Concrete values for data parameters are generated on-the-fly, i.e., during test execution, such that received outputs from the system can be taken into account for the inputs later provided in test execution. Due to undecidability of the satisfiability problem, SMT solvers may return result ‘unknown’. Our algorithm deals with this explicitly. We implemented our method in Maude combined with Z3, and use this to demonstrate the applicability of our method on the Bounded Retransmission Protocol benchmark. We measure performance by counting the number of inputs and outputs needed to discover bugs in mutants, i.e., in non-conforming variants of the specification. As a result, we find that we perform 3 times better, according to the geometric mean, than when using random testing as implemented by the tool TorXakis.

[1]  Mariëlle Stoelinga,et al.  Tester versus Bug: A Generic Framework for Model-Based Testing via Games , 2018, GandALF.

[2]  Alexandre Petrenko,et al.  Checking Experiments for Symbolic Input/Output Finite State Machines , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[3]  Bertrand Jeannet,et al.  Symbolic Test Selection Based on Approximate Analysis , 2005, TACAS.

[4]  Narciso Martí-Oliet,et al.  Maude: specification and programming in rewriting logic , 2002, Theor. Comput. Sci..

[5]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[6]  Nathan Kitchen Markov Chain Monte Carlo Stimulus Generation for Constrained Random Simulation , 2010 .

[7]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[8]  W. Eric Wong,et al.  Automatic test generation from communicating extended finite state machine (CEFSM)-based models , 2002, Proceedings Fifth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing. ISIRC 2002.

[9]  Frits W. Vaandrager,et al.  Proof-Checking a Data Link Protocol , 1994, TYPES.

[10]  Christophe Gaston,et al.  Symbolic Execution Techniques for Test Purpose Definition , 2006, TestCom.

[11]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs and Repetitive Quiescence , 1996, Softw. Concepts Tools.

[12]  Alan Hartman,et al.  Projected state machine coverage for software testing , 2002, ISSTA '02.

[13]  Nikolaj Bjørner,et al.  Alternating simulation and IOCO , 2011, International Journal on Software Tools for Technology Transfer.

[14]  Ramon Janssen,et al.  n-Complete Test Suites for IOCO , 2017, ICTSS.

[15]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[16]  Jan Tretmans,et al.  Test Generation Based on Symbolic Specifications , 2004, FATES.

[17]  Jan Tretmans,et al.  On the Existence of Practical Testers , 2017, ModelEd, TestEd, TrustEd.

[18]  Patrice Godefroid,et al.  SAGE: Whitebox Fuzzing for Security Testing , 2012, ACM Queue.

[19]  Mahesh A. Iyer,et al.  Race a word-level atpg-based constraints solver system for smart random simulation , 2003, International Test Conference, 2003. Proceedings. ITC 2003..

[20]  Cacm Staff,et al.  BufferBloat , 2011, Communications of the ACM.

[21]  Hartmut Ehrig,et al.  Fundamentals of Algebraic Specification 1: Equations and Initial Semantics , 1985 .

[22]  Jan Peleska,et al.  Complete model-based equivalence class testing for nondeterministic systems , 2016, Formal Aspects of Computing.

[23]  Jan Tretmans,et al.  A Symbolic Framework for Model-Based Testing , 2006, FATES/RV.