Bounded Reordering in the Distributed Test Architecture

In the distributed test architecture, the system under test (SUT) interacts with its environment at multiple physically distributed ports and the local testers at these ports do not synchronize their actions. This presents many challenges and, in particular, apparently incorrect behaviors can be the consequence of an erroneous assumption about the exact order in which actions were performed at different ports. In previous work, we defined a conformance relation for the distributed test architecture. Essentially, the SUT is faulty if we observe a trace <inline-formula> <tex-math notation="LaTeX">$\sigma$</tex-math></inline-formula> such that no admissible reordering of the actions in <inline-formula><tex-math notation="LaTeX">$\sigma$</tex-math></inline-formula> could have been produced by the specification. However, this notion can be weak if the compared traces might be <italic>too</italic> different. This paper introduces conformance relations where, for a given metric, a reordering is only considered if the distance between the two traces is at most a certain bound <inline-formula><tex-math notation="LaTeX">$k$</tex-math> </inline-formula>. We introduce two different metrics and provide algorithms to construct finite automata accepting these <italic>close</italic>, with respect to each metric, sequences. We also study the computational complexity of the two main problems associated with the new framework: deciding whether a trace is accepted by the new automaton and deciding whether one system conforms to a specification with respect to the new conformance relation.

[1]  Leslie Lamport,et al.  How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs , 2016, IEEE Transactions on Computers.

[2]  Robert M. Hierons,et al.  Generating Complete Controllable Test Suites for Distributed Testing , 2015, IEEE Transactions on Software Engineering.

[3]  Claude Jard,et al.  Testing Input/Output Partial Order Automata , 2007, TestCom/FATES.

[4]  Robert M. Hierons,et al.  Controllable Test Cases for the Distributed Test Architecture , 2008, ATVA.

[5]  Robert M. Hierons A More Precise Implementation Relation for Distributed Testing , 2016, Comput. J..

[6]  Robert M. Hierons,et al.  Timed implementation relations for the distributed test architecture , 2014, Distributed Computing.

[7]  Thierry Jéron,et al.  Towards Automatic Distribution of Testers for Distributed Conformance Testing , 1998, FORTE.

[8]  Robert M. Hierons,et al.  Implementation relations and test generation for systems with distributed interfaces , 2011, Distributed Computing.

[9]  Hasan Ural,et al.  The Synchronization Problem in Protocol Testing and its Complexity , 1991, Inf. Process. Lett..

[10]  Rajeev Alur,et al.  Model-checking of correctness conditions for concurrent objects , 1996, Proceedings 11th Annual IEEE Symposium on Logic in Computer Science.

[11]  Stefan Haar,et al.  Model-based testing for concurrent systems: unfolding-based test selection , 2014, International Journal on Software Tools for Technology Transfer.

[12]  Robert M. Hierons,et al.  Reaching and Distinguishing States of Distributed Systems , 2010, SIAM J. Comput..

[13]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[14]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[15]  Omar Rafiq,et al.  Coordination Algorithm for Distributed Testing , 2003, The Journal of Supercomputing.

[16]  Hasan Ural,et al.  Synchronizable test sequences based on multiple UIO sequences , 1995, TNET.

[17]  Ed Brinksma,et al.  Testing Real-Time Multi Input-Output Systems , 2005, ICFEM.

[18]  Antoni W. Mazurkiewicz,et al.  Traces, Histories, Graphs: Instances of a Process Monoid , 1984, International Symposium on Mathematical Foundations of Computer Science.

[19]  Hasan Ural,et al.  Constructing checking sequences for distributed testing , 2005, Formal Aspects of Computing.

[20]  Robert M. Hierons,et al.  Decidability and Complexity for Quiescent Consistency , 2016, 2016 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[21]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[22]  Gregor von Bochmann,et al.  Synchronization and Specification Issues in Protocol Testing , 1984, IEEE Trans. Commun..

[23]  Ana Sokolova,et al.  Quantitative relaxation of concurrent data structures , 2013, POPL.

[24]  Mary Jane Phillips-Matz,et al.  Traces , 1991 .

[25]  Ina Schieferdecker,et al.  Test Architectures for Distributed Systems: State of the Art and Beyond , 1998, IWTCS.

[26]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[27]  Stefan Haar,et al.  Distributed Testing of Concurrent Systems: Vector Clocks to the Rescue , 2014, ICTAC.

[28]  Ahmed Khoumsi,et al.  A Temporal Approach for Testing Distributed Systems , 2002, IEEE Trans. Software Eng..

[29]  Robert M. Hierons,et al.  The Effect of the Distributed Test Architecture on the Power of Testing , 2008, Comput. J..

[30]  Robert M. Hierons,et al.  UIO sequence based checking sequences for distributed test architectures , 2003, Inf. Softw. Technol..

[31]  Robert M. Hierons,et al.  Implementation Relations for the Distributed Test Architecture , 2008, TestCom/FATES.

[32]  Teruo Higashino,et al.  A survey on formal active and passive testing with applications to the cloud , 2015, Ann. des Télécommunications.

[33]  Omar Rafiq,et al.  Controllability and observability in distributed testing , 1999, Inf. Softw. Technol..

[34]  M. W. Shields An Introduction to Automata Theory , 1988 .

[35]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[36]  Gregor von Bochmann,et al.  Testing Systems Specified as Partial Order Input/Output Automata , 2008, TestCom/FATES.

[37]  Gregor von Bochmann,et al.  Error detection with multiple observers , 1985, PSTV.

[38]  Robert M. Hierons Oracles for Distributed Testing , 2012, IEEE Transactions on Software Engineering.

[39]  Ana R. Cavalli,et al.  A Framework for Distributed Testing of Timed Composite Systems , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[40]  Paul Pettersson,et al.  Chapter Three - A Research Overview of Tool-Supported Model-based Testing of Requirements-based Designs , 2015, Adv. Comput..

[41]  R. P. Dilworth,et al.  A DECOMPOSITION THEOREM FOR PARTIALLY ORDERED SETS , 1950 .

[42]  Stefan Haar,et al.  Unfolding-Based Test Selection for Concurrent Conformance , 2013, ICTSS.

[43]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[44]  Bruno Legeard,et al.  Model-based Testing: Where Does It Stand? , 2014, ACM Queue.

[45]  Yvan Labiche,et al.  A systematic review of state-based test tools , 2013, International Journal on Software Tools for Technology Transfer.

[46]  Robert M. Hierons Verifying and Comparing Finite State Machines for Systems that Have Distributed Interfaces , 2013, IEEE Transactions on Computers.

[47]  Gregor von Bochmann,et al.  Generating Synchronizable Test Sequences Based on Finite State Machine with Distributed Ports , 1993, Protocol Test Systems.