Increasing performance in byzantine fault-tolerant systems with on-demand replica consistency

Traditional agreement-based Byzantine fault-tolerant (BFT) systems process all requests on all replicas to ensure consistency. In addition to the overhead for BFT protocol and state-machine replication, this practice degrades performance and prevents throughput scalability. In this paper, we propose an extension to existing BFT architectures that increases performance for the default number of replicas by optimizing the resource utilization of their execution stages. Our approach executes a request on only a selected subset of replicas, using a selector component co-located with each replica. As this leads to divergent replica states, a selector on-demand updates outdated objects on the local replica prior to processing a request. Our evaluation shows that with each replica executing only a part of all requests, the overall performance of a Byzantine fault-tolerant NFS can be almost doubled; our prototype even outperforms unreplicated NFS.

[1]  Jason Flinn,et al.  Tolerating Latency in Replicated State Machines Through Client Speculation , 2009, NSDI.

[2]  Ben Y. Zhao,et al.  Awarded Best Student Paper! - Pond: The OceanStore Prototype , 2003 .

[3]  Marc Najork,et al.  Boxwood: Abstractions as the Foundation for Storage Infrastructure , 2004, OSDI.

[4]  Michael Burrows,et al.  The Chubby Lock Service for Loosely-Coupled Distributed Systems , 2006, OSDI.

[5]  Arun Venkataramani,et al.  Separating agreement from execution for byzantine fault tolerant services , 2003, SOSP '03.

[6]  Miguel Castro,et al.  Using abstraction to improve fault tolerance , 2001, Proceedings Eighth Workshop on Hot Topics in Operating Systems.

[7]  Miguel Castro,et al.  BASE: Using abstraction to improve fault tolerance , 2003, TOCS.

[8]  Rüdiger Kapitza,et al.  Hypervisor-Based Efficient Proactive Recovery , 2007, 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007).

[9]  Arun Venkataramani,et al.  ZZ and the art of practical BFT execution , 2011, EuroSys '11.

[10]  Miguel Oom Temudo de Castro,et al.  Practical Byzantine fault tolerance , 1999, OSDI '99.

[11]  Michael Williams,et al.  Replication in the harp file system , 1991, SOSP '91.

[12]  David E. Culler,et al.  Scalable, distributed data structures for internet service construction , 2000, OSDI.

[13]  William I. Nowicki,et al.  NFS: Network File System Protocol specification , 1989, RFC.

[14]  Michael G. Merideth Tradeoffs in Byzantine-Fault-Tolerant State-Machine-Replication Protocol Design , 2008 .

[15]  Ramakrishna Kotla,et al.  Zyzzyva: speculative byzantine fault tolerance , 2007, TOCS.

[16]  Michael J. Freedman,et al.  Prophecy: Using History for High-Throughput Fault Tolerance , 2010, NSDI.

[17]  Miguel Correia,et al.  How to tolerate half less one Byzantine nodes in practical distributed systems , 2004, Proceedings of the 23rd IEEE International Symposium on Reliable Distributed Systems, 2004..

[18]  Fred B. Schneider,et al.  Implementing fault-tolerant services using the state machine approach: a tutorial , 1990, CSUR.

[19]  Liuba Shrira,et al.  HQ replication: a hybrid quorum protocol for byzantine fault tolerance , 2006, OSDI '06.

[20]  Liuba Shrira,et al.  Providing high availability using lazy replication , 1992, TOCS.

[21]  Jacob R. Lorch,et al.  Farsite: federated, available, and reliable storage for an incompletely trusted environment , 2002, OSDI '02.

[22]  Sangmin Lee,et al.  Upright cluster services , 2009, SOSP '09.

[23]  Jehan-François Pâris,et al.  Voting with Witnesses: A Constistency Scheme for Replicated Files , 1986, ICDCS.

[24]  Leslie Lamport,et al.  Cheap Paxos , 2004, International Conference on Dependable Systems and Networks, 2004.

[25]  Mahadev Konar,et al.  ZooKeeper: Wait-free Coordination for Internet-scale Systems , 2010, USENIX ATC.

[26]  Ramakrishna Kotla,et al.  High throughput Byzantine fault tolerance , 2004, International Conference on Dependable Systems and Networks, 2004.

[27]  Jeffrey Katcher,et al.  PostMark: A New File System Benchmark , 1997 .

[28]  Gary Carpenter 동적 사용자를 위한 Scalable 인증 그룹 키 교환 프로토콜 , 2005 .

[29]  Michael K. Reiter,et al.  Low-overhead byzantine fault-tolerant storage , 2007, SOSP.

[30]  Tobias Distler,et al.  SPARE: Replicas on Hold , 2011, NDSS.

[31]  Robbert van Renesse,et al.  Chain Replication for Supporting High Throughput and Availability , 2004, OSDI.

[32]  Brett D. Fleisch,et al.  The Chubby lock service for loosely-coupled distributed systems , 2006, OSDI '06.

[33]  Michael K. Reiter,et al.  Fault-scalable Byzantine fault-tolerant services , 2005, SOSP '05.

[34]  Ben Y. Zhao,et al.  Pond: The OceanStore Prototype , 2003, FAST.