Passive testing of communicating systems with timeouts

ContextThe design of complex systems demands methodologies to analyze its correct behaviour. It is usual that a correct behaviour is determined by the compliance with temporal requirements. Currently, testing is the most used technology to validate the correctness of systems. Although several techniques that take into account time aspects have been proposed, most of them require the tester interacts with the system. However, if this is not possible, it is necessary to apply a passive testing approach where the tester monitors the behaviour of the system. ObjectiveThe aim of this paper is to propose a methodology to perform passive testing on communicating systems in which the behaviour of their components must fulfill temporal restrictions associated with both performance and delays/timeouts. MethodOur framework uses algorithms for checking traces collected from the systems against invariants which formally represent the most relevant properties that must be fulfilled by the system. In order to support the feasibility of the methodology, we have performed an empirical study on a complex system for automatic recognition of images based on a pipeline architecture. We have analyzed the correctness of the system's behaviour with respect to a set of invariants. Finally, an experiment, based on mutations of the system, was conducted to study the level of detection of a set of invariants. ResultsDifferent errors were detected and fixed along the development of the system by means of the proposed methodology. The results of the experiments with the mutated versions of the system indicated that the designed set of invariants was more effective in finding errors associated to temporal aspects than those related to communication among components. ConclusionThe proposed technique has been shown to be very useful for analyzing complex timed systems, and find errors when the tester has no control over their behaviour.

[1]  W. Eric Wong,et al.  Mutation Testing Applied to Validate SDL Specifications , 2004, TestCom.

[2]  Jesus Carretero,et al.  Using Architectural Simulation Models to Aid the Design of Data Intensive Application , 2009, 2009 Third International Conference on Advanced Engineering Computing and Applications in Sciences.

[3]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[4]  Mercedes G. Merayo,et al.  Extending EFSMs to Specify and Test Timed Systems with Action Durations and Time-Outs , 2008, IEEE Transactions on Computers.

[5]  Marian Gheorghe,et al.  Communicating Stream X-Machines Systems are no more than X-Machines , 1999, J. Univers. Comput. Sci..

[6]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[7]  Victor Mitrana,et al.  Parallel Finite Automata Systems Communicating by States , 2002, Int. J. Found. Comput. Sci..

[8]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[9]  Mercedes G. Merayo,et al.  Formal passive testing of timed systems: theory and tools , 2012, Softw. Test. Verification Reliab..

[10]  A. J. Offutt.  A practical system for mutation testing: help for the common programmer , 1994, Proceedings., International Test Conference.

[11]  Mercedes G. Merayo,et al.  Testing Finite State Machines Presenting Stochastic Time and Timeouts , 2007, EPEW.

[12]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[13]  Michael Merritt,et al.  Time-Constrained Automata (Extended Abstract) , 1991, CONCUR.

[14]  Thierry Jéron,et al.  Remote testin can be as powerful as local testing , 1999, FORTE.

[15]  Kim G. Larsen,et al.  Testing Real-Time Systems Using UPPAAL , 2008, Formal Methods and Testing.

[16]  P. M. Melliar-Smith,et al.  Synchronizing clocks in the presence of faults , 1985, JACM.

[17]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[18]  Mercedes G. Merayo,et al.  Extending EFSMs to Specify and Test Timed Systems with Action Durations and Timeouts , 2006, FORTE.

[19]  Jesús Carretero,et al.  New techniques for simulating high performance MPI applications on large storage networks , 2008, 2008 IEEE International Conference on Cluster Computing.

[20]  Nancy A. Lynch,et al.  The Theory of Timed I/O Automata (Synthesis Lectures in Computer Science) , 2006 .

[21]  Roberto Barbuti,et al.  Timed automata with urgent transitions , 2004, Acta Informatica.

[22]  Leszek Wojnar,et al.  An Automatic Recognition of the Frontal Sinus in X-Ray Images of Skull , 2009, IEEE Transactions on Biomedical Engineering.

[23]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[24]  Robert M. Hierons,et al.  Testing timed systems modeled by Stream X-machines , 2011, Software & Systems Modeling.

[25]  Daniel Brand,et al.  On Communicating Finite-State Machines , 1983, JACM.

[26]  Mercedes G. Merayo,et al.  Passive Testing of Stochastic Timed Systems , 2009, 2009 International Conference on Software Testing Verification and Validation.

[27]  Stéphane Maag,et al.  Passive performance testing of network protocols , 2014, Comput. Commun..

[28]  Samrat S. Batth,et al.  Fault masking by multiple timing faults in timed EFSM models , 2009, Comput. Networks.

[29]  William Gropp,et al.  Mpi---the complete reference: volume 1 , 1998 .

[30]  Jesús Carretero,et al.  SIMCAN: A flexible, scalable and expandable simulation platform for modelling and simulating distributed architectures and applications , 2012, Simul. Model. Pract. Theory.

[31]  Emal Pasarly.  Time , 2011, Encyclopedia of Evolutionary Psychological Science.

[32]  Joseph Sifakis,et al.  Modeling Urgency in Timed Systems , 1997, COMPOS.

[33]  Nina Yevtushenko,et al.  Distinguing Non-deterministic Timed Finite State Machines , 2009, FMOODS/FORTE.

[34]  Frits W. Vaandrager,et al.  Testing timed automata , 1997, Theor. Comput. Sci..

[35]  Mercedes G. Merayo,et al.  Formal testing from timed finite state machines , 2008, Comput. Networks.

[36]  Florentin Ipate.  Test Selection for Hierarchical and Communicating Finite State Machines , 2009, Comput. J..

[37]  Stavros Tripakis,et al.  An Expressive and Implementable Formal Framework for Testing Real-Time Systems , 2005, TestCom.

[38]  J. Barnard,et al.  Communicating X-machines , 1996, Inf. Softw. Technol..

[39]  Mercedes G. Merayo.  Passive Testing of Timed Systems with Timeouts , 2012, 2012 12th International Conference on Quality Software.

[40]  Nancy A. Lynch,et al.  The Theory of Timed I/o Automata , 2003 .

[41]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[42]  David Lee,et al.  Conformance testing of protocols specified as communicating finite state machines-a guided random walk based approach , 1996, IEEE Trans. Commun..

[43]  Julien Schmaltz,et al.  On Conformance Testing for Timed Systems , 2008, FORMATS.

[44]  Robert M. Hierons,et al.  Mutation Testing , 2010, Encyclopedia of Software Engineering.

[45]  Rajeev Alur,et al.  A Theory of Timed Automata , 1994, Theor. Comput. Sci..

[46]  Biniam Gebremichael,et al.  Specifying urgency in timed I/O automata , 2005, Third IEEE International Conference on Software Engineering and Formal Methods (SEFM'05).