The complexity of asynchronous model based testing

In model based testing (MBT), testing is based on a model M that typically is expressed using a state-based language such as an input output transition system (IOTS). Most approaches to MBT assume that communications between the system under test (SUT) and its environment are synchronous. However, many systems interact with their environment through asynchronous channels and the presence of such channels changes the nature of testing. In this paper we investigate the situation in which the SUT interacts with its environment through asynchronous channels and the problems of producing test cases to reach a state, execute a transition, or to distinguish two states. In addition, we investigate the Oracle Problem. All four problems are explored for both FIFO and non-FIFO channels. It is known that the Oracle Problem can be solved in polynomial time for FIFO channels but we also show that the three test case generation problems can also be solved in polynomial time in the case where the IOTS is observable but the general test generation problems are EXPTIME-hard. For non-FIFO channels we prove that all of the test case generation problems are EXPTIME-hard and the Oracle Problem in NP-hard, even if we restrict attention to deterministic IOTSs.

[1]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[2]  Sagar Naik,et al.  Generation of Adaptive Test Cases from Nondeterministic Finite State Models , 1992, Protocol Test Systems.

[3]  Alfred V. Aho,et al.  An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours , 1991, IEEE Trans. Commun..

[4]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[5]  David Harel,et al.  Modeling Reactive Systems With Statecharts : The Statemate Approach , 1998 .

[6]  G. Bochmann,et al.  Testing deterministic implementations from nondeterministic FSM specifications , 1996 .

[7]  Wolfgang Grieskamp,et al.  Multi-paradigmatic Model-Based Testing , 2006, FATES/RV.

[8]  John H. Reif,et al.  The Complexity of Two-Player Games of Incomplete Information , 1984, J. Comput. Syst. Sci..

[9]  Alexandre Petrenko,et al.  Test Selection Based on Communicating Nondeterministic Finite-State Machines Using a Generalized WP-Method , 1994, IEEE Trans. Software Eng..

[10]  Margus Veanes,et al.  Towards a Tool Environment for Model-Based Testing with AsmL , 2003, FATES.

[11]  Robert M. Hierons,et al.  Adaptive Testing of a Deterministic Implementation Against a Nondeterministic Finite State Machine , 1998, Comput. J..

[12]  Franz Wotawa,et al.  Asynchronous Input-Output Conformance Testing , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[13]  Claude Jard,et al.  Testing Input/Output Partial Order Automata , 2007, TestCom/FATES.

[14]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[15]  Robert M. Hierons.  Applying adaptive test cases to nondeterministic implementations , 2006, Inf. Process. Lett..

[16]  Wolfgang Grieskamp,et al.  Model‐based quality assurance of protocol documentation: tools and methodology , 2011, Softw. Test. Verification Reliab..

[17]  Nina Yevtushenko,et al.  Testing Transition Systems with Input and Output Testers , 2003, TestCom.

[18]  Rajeev Alur,et al.  Distinguishing tests for nondeterministic and probabilistic machines , 1995, STOC '95.

[19]  F. C. Hennie.  Fault detecting experiments for sequential circuits , 1964, SWCT.

[20]  Jan Tretmans,et al.  On Asynchronous Testing , 1992, Protocol Test Systems.

[21]  Thierry Jéron,et al.  Towards Automatic Distribution of Testers for Distributed Conformance Testing , 1998, FORTE.

[22]  Gregor von Bochmann,et al.  Testing Systems Specified as Partial Order Input/Output Automata , 2008, TestCom/FATES.

[23]  Thierry Jéron,et al.  Remote testin can be as powerful as local testing , 1999, FORTE.

[24]  Olaf Owe,et al.  Model Testing Asynchronously Communicating Objects using Modulo AC Rewriting , 2010, Electron. Notes Theor. Comput. Sci..

[25]  Margus Veanes,et al.  Generating finite state machines from abstract state machines , 2002, ISSTA '02.

[26]  Robert M. Hierons,et al.  Testing from a nondeterministic finite state machine using adaptive state counting , 2004, IEEE Transactions on Computers.

[27]  Alan Hartman,et al.  Using a model-based test generator to test for standard conformance , 2002, IBM Syst. J..

[28]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[29]  Robert M. Hierons,et al.  Optimizing the length of checking sequences , 2006, IEEE Transactions on Computers.

[30]  Matthew Hennessy,et al.  Testing Theories for Asynchronous Languages , 1998, FSTTCS.

[31]  Robert E. Tarjan,et al.  Depth-First Search and Linear Graph Algorithms , 1972, SIAM J. Comput..

[32]  Rocco De Nicola,et al.  Asynchronous Observations of Processes , 1998, FoSSaCS.

[33]  Alexandre Petrenko,et al.  Transition covering tests for systems with queues , 2009, Softw. Test. Verification Reliab..

[34]  Alexandre Petrenko,et al.  On Testing Partially Specified IOTS through Lossless Queues , 2004, TestCom.

[35]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[36]  Rocco De Nicola,et al.  A Theory of "May" Testing for Asynchronous Languages , 1999, FoSSaCS.

[37]  Dana S. Scott,et al.  Finite Automata and Their Decision Problems , 1959, IBM J. Res. Dev..

[38]  A. Jefferson Offutt,et al.  Generating Tests from UML Specifications , 1999, UML.

[39]  Robert M. Hierons,et al.  Reaching and Distinguishing States of Distributed Systems , 2010, SIAM J. Comput..

[40]  Thomas J. Schaefer,et al.  The complexity of satisfiability problems , 1978, STOC.

[41]  Hasan Ural,et al.  A test generation algorithm for systems modelled as non-deterministic FSMs , 1993, Softw. Eng. J..

[42]  Paul Gastin,et al.  A Fresh Look at Testing for Asynchronous Communication , 2006, ATVA.

[43]  Alexandre Petrenko,et al.  Generating asynchronous test cases from test purposes , 2011, Inf. Softw. Technol..

[44]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[45]  Fan Zhang,et al.  Optimal Transfer Trees and Distinguishing Trees for Testing Observable Nondeterministic Finite-State Machines , 2003, IEEE Trans. Software Eng..

[46]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[47]  Tsong Yueh Chen,et al.  Fault-based testing in the absence of an oracle , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[48]  Nina Yevtushenko,et al.  Nondeterministic State Machines in Protocol Conformance Testing , 1993, Protocol Test Systems.