Structural Complexity and Execution Efficiency of Distributed Application Protocols

The complexity and efficiency aspects of a distributed application protocol (e.g., replicated data access) are often intertwined with the application-specific communication requirements between protocol entities and the underlying primitives for message broadcasting. For instance, the choice between recovery by application from a mis-ordered message delivery versus providing message ordering property in the communication layer (CL) is based on tradeoffs between protocol complexity and efficiency. The paper systematically analyzes these tradeoffs with respect to message ordering and atomicity properties in broadcast communication. Enforcing these properties in the CL leads to uniform communication structure of applications; however, the CL incurs message and execution time overhead due to extensive communication state (e.g., message buffering). When these properties are not supported by CL but are required for an application, mechanisms to enforce it are built into the application protocol, thereby increasing complexity in the protocol structure. Detailed examples of applications are given to illustrate these tradeoffs. The analysis is useful in designing communication primitives and application protocols for distributed systems.