Choosing Between Fault-Tolerance and Increased V&V for Improving Reliability

Fault tolerant systems based on the use of software design diversity may be able to achieve high levels of reliability more cost-effectively than other approaches, such as heroic debugging. Earlier experiments have shown that multi-version software systems are more reliable than the individual versions. However, it is also clear that the reliability benefits are much worse than would be suggested by naive assumptions of failure independence between the versions. To decide whether to use design diversity or other means for achieving the desired reliability a developer would need to know how they compare from the viewpoint of cost-effectiveness. Empirical data are insufficient for deciding this question, and expert opinions differ. We refute a recently published argument in favour of diversity and in the process show some general factors deciding whether process improvement, or debugging of the versions in a multiple-version system, will increase or decrease the statistical correlation between failures of the versions. The conclusion is that there is as yet no evidence that the choice between design diversity and other means of reliability improvement can be decided by general arguments rather than by detailed (and uncertain) special-case analysis.