Lead-time Impact of Distributed Testing on Distributed Real-time Systems

Testing of distributed real-time systems (DRTSs) is an expensive and time consuming activity. This master thesis investigates how lead-time can be reduced when developing DRTSs. The concept of distributed testing is introduced. Distributed testing takes a holistic view of the distributed system rather than viewing the individual components as separate entities. An introduction to the domain of testing and DRTS in general is given. Testing is a continuous activity throughout the entire development process. This work focuses on late testing phases. In a DRTS many issues, not present in non-distributed systems, need to be considered. Real-time constraints co-exist with communication lines that may be unreliable and nodes that may occasionally break down. When testing DRTSs, the main issues are the collection of timing data and minimization of the probe effect. This master thesis suggests a version of vector clocks as a way of relating events across a DRTS, and discusses how to elicit which information to monitor when testing. A prototype implementation of a distributed testing tool is suggested, based on the theoretical discussion. The software system used for the implementation is a state-based system using a message passing protocol for communication. The prototype implementation shows that it is feasible to implement a tool supporting distributed testing. However, there are many implementation and testing issues that need to be resolved to minimize perturbation. These issues are discussed in the report but are not part of the prototype. To concretize the benefits of using distributed testing, an empiric experiment investigates the effectiveness of distributed testing. The experiment evaluates distributed testing compared to non-distributed testing. Participants in the experiment are sampled from both academic and industrial groups. From the result, it is concluded that the benefits of distributed testing are clear, with good statistical significance and validity threats under control. Distributed systems are becoming increasingly popular. For developers of CASE (Computer Aided Software Engineering) and CAST (Computer Aided Software Testing) tools, this master thesis can provide insight and sales argument for developing distributed testing tools. Also, for companies developing DRTSs, many important issues for purchasing testing tools are presented in this report. Perhaps most important, the experiment show that money invested in distributed testing is likely to be returned as the time spent on testing decreases.