The Need for Realistic Failure Models in Protocol Design

Fault tolerant algorithms are often designed under the assumption that no more than t out of n processes or components can fail. This approach was pioneered by the SIFT project [22], and has since been widely applied to the design of algorithms for real critical systems, e.g., air traffic control [6], other highly available services like file servers [15], and so on. It is such a common assumption that most fault tolerant algorithms found in the literature today adopt it without any justification (e.g., [14, 19]). It is a common assumption because the t out of n model gives one a simple abstraction for reasoning about failure-prone environments and system reliability. With this assumption it is fairly easy to design and verify protocols and also to express lower and upper bounds. Unfortunately, when adopting this assumption, we often forget the relationship between the t out of n assumption and system reliability. In real systems reliability is typically expressed in terms of the probability that the system meets its specification. A more refined model expresses survivability as a range of probabilities for the system meeting a number of different degraded specifications [13]. When characterizing the possible failures as t out of n, one is implicitly expressing an upper bound on system reliability as the probability that t failures or less occur throughout the time the algorithm runs. Forgetting the relation between the t out of n assumption and system reliability can lead to a foolish design. For example, some consensus protocols (e.g., [7]) have a structure in which once t failures have been detected the protocol proceeds with the assumption that no further failures will occur. A more sensible design would have the protocol become more cautious under these circumstances: if t failures have occurred then it is possible that the failure analysis done in computing t was faulty, and so further failures may be more likely under these circumstances. Such foolishness can occur in a more subtle manner. In this position paper we remind ourselves of the relation between failure assumptions and system reliability. We bring to the surface some implicit assumptions made by the t out of n failure characterization and discuss their limitations. We briefly discuss failure assumptions that address some of these limitations. We argue that not all the shortcomings of this model have adequate solutions as of yet.