Testing nondeterminate systems

The behavior of nondeterminate systems can be hard to predict, since similar inputs at different times can generate different outputs. In other words, the behavior seen during the testing process may not be seen at runtime. Due to the uncertainties associated with nondeterminism, the standard view is that we should avoid such nondeterminate systems, especially for systems requiring high reliability. While this is a valid guideline, at least in two application areas such nondeterminacy is unavoidable. Early life-cycle requirements and AI software are becoming widely used, yet both are imprecise and may exhibit nondeterminate behaviour if explored rigorously by a test device. Based on a literature review and some theoretical studies, we argue that many stable properties exist within the space of all possible nondeterminate behaviors. However, we also show that seemingly trivial changes to a nondeterministic system can turn an easily testable system into an impossibly hard system to test. Finally, we stress that this analysis does not imply a correlation between stable zones of nondeterminate testability and the ultimate maintainability of nondeterminate systems. That is, while we are optimistic about testing nondeterminate systems, we remain cautious about the maintenance of such systems.