Combining Model-Based Testing and Runtime Monitoring for Program Testing in the Presence of Nondeterminism

In case of underspecified or not fully predictable systems, models specifying system behaviors are nondeterministic. Nondeterminism poses several challenges for the validation and verification activities, including the problem of inconclusive tests in model-based testing with model checker. It is a validation technique that uses model checker counterexamples as test cases. In this paper, we tackle the problem of testing nondeterministic systems by combining model-based testing and runtime conformance monitoring: the input sequences of the tests are automatically generated from nondeterministic models; then their execution is runtime monitored to check conformance of the code w.r.t. its specification. This technique provides an oracle for the test data, it never bears inconclusive responses, and it allows measuring the requirement coverage. The approach uses the Abstract State Machines as formal method for specification purposes and Java as implementation language. As a proof of concepts, the Tic-Tac-Toe game is taken as example of a system with nondeterministic behavior (both at specification and code levels).

[1]  Jan Tretmans,et al.  Model Based Testing with Labelled Transition Systems , 2008, Formal Methods and Testing.

[2]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[3]  Rajeev Alur,et al.  Distinguishing tests for nondeterministic and probabilistic machines , 1995, STOC '95.

[4]  Jan Tretmans,et al.  Testing Transition Systems: An Annotated Bibliography , 2000, MOVEP.

[5]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[6]  Michael R. Lowry,et al.  Combining test case generation and runtime verification , 2005, Theor. Comput. Sci..

[7]  Paolo Arcaini,et al.  CoMA: Conformance Monitoring of Java Programs by Abstract State Machines , 2011, RV.

[8]  Paul A. Strooper,et al.  Dealing with non-determinism in testing concurrent Java components , 2005, 12th Asia-Pacific Software Engineering Conference (APSEC'05).

[9]  Angelo Gargantini,et al.  An Evaluation of Model Checkers for Specification Based Test Case Generation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[10]  Egon Börger,et al.  Abstract State Machines. A Method for High-Level System Design and Analysis , 2003 .

[11]  David F. Bacon,et al.  Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion , 2007, OOPSLA 2007.

[12]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[13]  Margus Veanes,et al.  Online testing with model programs , 2005, ESEC/FSE-13.

[14]  Jan Tretmans,et al.  Test Generation with Inputs, Outputs, and Quiescence , 1996, TACAS.

[15]  Angelo Gargantini,et al.  Using Spin to Generate Testsfrom ASM Specifications , 2003, Abstract State Machines.

[16]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[17]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.

[18]  Angelo Gargantini,et al.  ASM-Based Testing: Coverage Criteria and Automatic Test Sequence , 2001, J. Univers. Comput. Sci..

[19]  Gerard J. Holzmann,et al.  The SPIN Model Checker , 2003 .

[20]  Robert M. Hierons,et al.  Testing from a nondeterministic finite state machine using adaptive state counting , 2004, IEEE Transactions on Computers.

[21]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[22]  Nina Yevtushenko,et al.  Generating Checking Sequences for Nondeterministic Finite State Machines , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[23]  Alexandre Petrenko,et al.  Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography , 2000, MOVEP.

[24]  Gordon Fraser,et al.  Nondeterministic Testing with Linear Model-Checker Counterexamples , 2007 .

[25]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[26]  Gordon Fraser,et al.  Test-Case Generation and Coverage Analysis for Nondeterministic Systems Using Model-Checkers , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[27]  Jan Tretmans,et al.  A Symbolic Framework for Model-Based Testing , 2006, FATES/RV.

[28]  Alexandre Petrenko,et al.  Can a Model Checker Generate Tests for Non-Deterministic Systems? , 2007, MBT.

[29]  Pieter Bekaert,et al.  Non-determinism in conceptual models , 2001, OOPSLA 2001.