An Architecture for Survivable Coordination in Large Distributed Systems

Coordination among processes in a distributed system can be rendered very complex in a large-scale system where messages may be delayed or lost and when processes may participate only transiently or behave arbitrarily, e.g. after suffering a security breach. In this paper, we propose a scalable architecture to support coordination in such extreme conditions. Our architecture consists of a collection of persistent data servers that implement simple shared data abstractions for clients, without trusting the clients or even the servers themselves. We show that, by interacting with these untrusted servers, clients can solve distributed consensus, a powerful and fundamental coordination primitive. Our architecture is very practical, and we describe the implementation of its main components in a system called Fleet.

[1]  Martín Abadi,et al.  Authentication in distributed systems: theory and practice , 1991, SOSP '91.

[2]  Cynthia Dwork,et al.  Randomization in Byzantine Agreement , 1989, Adv. Comput. Res..

[3]  Avishai Wool,et al.  Quorum Systems in Replicated Databases: Science or Fiction? , 1998, IEEE Data Eng. Bull..

[4]  Robert H. Thomas,et al.  A Majority consensus approach to concurrency control for multiple copy databases , 1979, ACM Trans. Database Syst..

[5]  Miguel Oom Temudo de Castro,et al.  Practical Byzantine fault tolerance , 1999, OSDI '99.

[6]  Maurice Herlihy,et al.  Fast Randomized Consensus Using Shared Memory , 1990, J. Algorithms.

[7]  Paul Hudak,et al.  Memory coherence in shared virtual memory systems , 1989, TOCS.

[8]  Avishai Wool,et al.  Probabilistic Byzantine quorum systems , 1998, PODC '98.

[9]  LamportLeslie.  Time, clocks, and the ordering of events in a distributed system , 1978 .

[10]  Nancy A. Lynch,et al.  Impossibility of distributed consensus with one faulty process , 1985, JACM.

[11]  Michael K. Reiter,et al.  Secure agreement protocols: reliable and atomic group multicast in rampart , 1994, CCS '94.

[12]  Tal Rabin,et al.  Secure distributed storage and retrieval , 1997, Theor. Comput. Sci..

[13]  Yishay Mansour,et al.  On diffusing updates in a Byzantine environment , 1999, Proceedings of the 18th IEEE Symposium on Reliable Distributed Systems.

[14]  Willy Zwaenepoel,et al.  Munin: distributed shared memory based on type-specific memory coherence , 1990, PPOPP '90.

[15]  Fred B. Schneider,et al.  Implementing fault-tolerant services using the state machine approach: a tutorial , 1990, CSUR.

[16]  B SchneiderFred.  Implementing fault-tolerant services using the state machine approach: a tutorial , 1990 .

[17]  Miguel Castro,et al.  Safe and efficient sharing of persistent objects in Thor , 1996, SIGMOD '96.

[18]  Yvo Desmedt,et al.  Shared Generation of Authenticators and Signatures (Extended Abstract) , 1991, CRYPTO.

[19]  Paul Hudak,et al.  Memory coherence in shared virtual memory systems , 1986, PODC '86.

[20]  Hagit Attiya,et al.  Sharing memory robustly in message-passing systems , 1990, PODC '90.

[21]  Paul D. Ezhilchelvan,et al.  Principal Features of the VOLTAN Family of Reliable Node Architectures for Distributed Systems , 1992, IEEE Trans. Computers.

[22]  Louise E. Moser,et al.  The SecureRing protocols for securing group communication , 1998, Proceedings of the Thirty-First Hawaii International Conference on System Sciences.

[23]  Moti Yung,et al.  How to share a function securely , 1994, STOC '94.

[24]  Nancy A. Lynch,et al.  Impossibility of distributed consensus with one faulty process , 1983, PODS '83.

[25]  B. Lampson,et al.  Authentication in distributed systems: theory and practice , 1991, TOCS.

[26]  Michael K. Reiter,et al.  Secure and scalable replication in Phalanx , 1998, Proceedings Seventeenth IEEE Symposium on Reliable Distributed Systems (Cat. No.98CB36281).

[27]  Morrie Gasser,et al.  An architecture for practical delegation in a distributed system , 1990, Proceedings. 1990 IEEE Computer Society Symposium on Research in Security and Privacy.

[28]  R. Anderson.  The Eternity Service , 1996 .

[29]  Adi Shamir,et al.  A method for obtaining digital signatures and public-key cryptosystems , 1978, CACM.

[30]  Michael K. Reiter,et al.  Byzantine quorum systems , 1997, STOC '97.

[31]  Avishai Wool,et al.  Probabilistic Quorum Systems , 2001, Inf. Comput..

[32]  Avishai Wool,et al.  The load and availability of Byzantine quorum systems , 1997, PODC '97.

[33]  Miguel Castro,et al.  A Correctness Proof for a Practical Byzantine-Fault-Tolerant Replication Algorithm , 1999 .

[34]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[35]  Leslie Lamport,et al.  The Byzantine Generals Problem , 1982, TOPL.

[36]  Matthew K. Franklin,et al.  The Ω key management service , 1996, CCS '96.

[37]  Michael K. Reiter,et al.  Probabilistic quorum systems , 1997, PODC '97.

[38]  Michael K. Reiter,et al.  Survivable consensus objects , 1998, Proceedings Seventeenth IEEE Symposium on Reliable Distributed Systems (Cat. No.98CB36281).

[39]  Adi Shamir,et al.  A method for obtaining digital signatures and public-key cryptosystems , 1978, CACM.

[40]  Michael K. Reiter,et al.  Fault detection for Byzantine quorum systems , 1999, Dependable Computing for Critical Applications 7.

[41]  Whitfield Diffie,et al.  New Directions in Cryptography , 1976, IEEE Trans. Inf. Theory.

[42]  Barbara Liskov,et al.  Implementation of Argus , 1987, SOSP '87.

[43]  David K. Gifford,et al.  Weighted voting for replicated data , 1979, SOSP '79.

[44]  P. V. McMahon.  SESAME V2 public key and authorisation extensions to Kerberos , 1995, Proceedings of the Symposium on Network and Distributed System Security.

[45]  Morrie Gasser,et al.  The Digital Distributed System Security Architecture , 1989 .

[46]  Matthew K. Franklin,et al.  The Omega Key Management Service , 1996, J. Comput. Secur..

[47]  Brian A. Coan,et al.  A Simple and Efficient Randomized Byzantine Agreement Algorithm , 1985, IEEE Transactions on Software Engineering.