Troxy: Transparent Access to Byzantine Fault-Tolerant Systems

Various protocols and architectures have been proposed to make Byzantine fault tolerance (BFT) increasingly practical. However, the deployment of such systems requires dedicated client-side functionality. This is necessary as clients have to connect to multiple replicas and perform majority voting over the received replies to outvote faulty responses. Deploying custom client-side code is cumbersome, and often not an option, especially in open heterogeneous systems and for well-established protocols (e.g., HTTP and IMAP) where diverse client-side implementations co-exist. We propose Troxy, a system which relocates the BFT-specific client-side functionality to the server side, thereby making BFT transparent to legacy clients. To achieve this, Troxy relies on a trusted subsystem built upon hardware protection enabled by Intel SGX. Additionally, Troxy reduces the replication cost of BFT for read-heavy workloads by offering an actively maintained cache that supports trustworthy read operations while preserving the consistency guarantees offered by the underlying BFT protocol. A prototype of Troxy has been built and evaluated, and results indicate that using Troxy (1) leads to at most 43% performance loss with small ordered messages in a local network environment, while (2) improves throughput by 130% with read-heavy workloads in a simulated wide-area network.

[1]  Wenbing Zhao,et al.  BFT-WS: A Byzantine Fault Tolerance Framework for Web Services , 2007, 2007 Eleventh International IEEE EDOC Conference Workshop.

[2]  Marcus Peinado,et al.  T-SGX: Eradicating Controlled-Channel Attacks Against Enclave Programs , 2017, NDSS.

[3]  Tobias Distler,et al.  SPARE: Replicas on Hold , 2011, NDSS.

[4]  Ramakrishna Kotla,et al.  Zyzzyva: speculative byzantine fault tolerance , 2007, TOCS.

[5]  Ramakrishna Kotla,et al.  High throughput Byzantine fault tolerance , 2004, International Conference on Dependable Systems and Networks, 2004.

[6]  Alysson Bessani,et al.  A Byzantine Fault-Tolerant Ordering Service for the Hyperledger Fabric Blockchain Platform , 2017, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[7]  Tobias Distler,et al.  Resource-Efficient Byzantine Fault Tolerance , 2016, IEEE Transactions on Computers.

[8]  John K. Ousterhout,et al.  In Search of an Understandable Consensus Algorithm , 2014, USENIX ATC.

[9]  David M. Eyers,et al.  SCONE: Secure Linux Containers with Intel SGX , 2016, OSDI.

[10]  Priya Narasimhan,et al.  Thema: Byzantine-fault-tolerant middleware for Web-service applications , 2005, 24th IEEE Symposium on Reliable Distributed Systems (SRDS'05).

[11]  Rüdiger Kapitza,et al.  TrustJS: Trusted Client-side Execution of JavaScript , 2017, EUROSEC.

[12]  Johannes Behl,et al.  Hybrids on Steroids: SGX-Based High Performance BFT , 2017, EuroSys.

[13]  Hovav Shacham,et al.  Iago attacks: why the system call API is a bad untrusted RPC interface , 2013, ASPLOS '13.

[14]  Christof Fetzer,et al.  SecureKeeper: Confidential ZooKeeper using Intel SGX , 2016, Middleware.

[15]  Insik Shin,et al.  SGX-Shield: Enabling Address Space Layout Randomization for SGX Programs , 2017, NDSS.

[16]  Scott Shenker,et al.  Attested append-only memory: making adversaries stick to their word , 2007, SOSP.

[17]  Rüdiger Kapitza,et al.  AsyncShock: Exploiting Synchronisation Bugs in Intel SGX Enclaves , 2016, ESORICS.

[18]  Adrian Perrig,et al.  TrustVisor: Efficient TCB Reduction and Attestation , 2010, 2010 IEEE Symposium on Security and Privacy.

[19]  Ju Wang,et al.  Windows Azure Storage: a highly available cloud storage service with strong consistency , 2011, SOSP.

[20]  Miguel Oom Temudo de Castro,et al.  Practical Byzantine fault tolerance , 1999, OSDI '99.

[21]  Marcus Peinado,et al.  Inferring Fine-grained Control Flow Inside SGX Enclaves with Branch Shadowing , 2016, USENIX Security Symposium.

[22]  Johannes Behl,et al.  CheapBFT: resource-efficient byzantine fault tolerance , 2012, EuroSys '12.

[23]  Marko Vukolic,et al.  Gracefully Degrading Fair Exchange with Security Modules , 2005, EDCC.

[24]  Miguel Correia,et al.  An Intrusion-Tolerant Web Server based on the DISTRACT Architecture , 2004 .

[25]  Jacob R. Lorch,et al.  TrInc: Small Trusted Hardware for Large Distributed Systems , 2009, NSDI.

[26]  Arun Venkataramani,et al.  Separating agreement from execution for byzantine fault tolerant services , 2003, SOSP '03.

[27]  Michael Dahlin,et al.  Making Byzantine Fault Tolerant Systems Tolerate Byzantine Faults , 2009, NSDI.

[28]  Tobias Distler,et al.  Increasing performance in byzantine fault-tolerant systems with on-demand replica consistency , 2011, EuroSys '11.

[29]  Miguel Correia,et al.  Efficient Byzantine Fault-Tolerance , 2013, IEEE Transactions on Computers.

[30]  E. Sirer,et al.  Scaling Databases through Trusted Hardware Proxies , 2017, SysTEX@SOSP.

[31]  Emmett Witchel,et al.  Ryoan: A Distributed Sandbox for Untrusted Computation on Secret Data , 2016, OSDI.

[32]  Ittai Anati,et al.  Innovative Technology for CPU Based Attestation and Sealing , 2013 .

[33]  Nuno Ferreira Neves,et al.  Securing Passive Replication through Verification , 2015, 2015 IEEE 34th Symposium on Reliable Distributed Systems (SRDS).

[34]  Michael J. Freedman,et al.  Prophecy: Using History for High-Throughput Fault Tolerance , 2010, NSDI.

[35]  Leslie Lamport,et al.  The part-time parliament , 1998, TOCS.

[36]  Johannes Behl,et al.  Consensus-Oriented Parallelization: How to Earn Your First Million , 2015, Middleware.

[37]  Miguel Castro,et al.  Proactive recovery in a Byzantine-fault-tolerant system , 2000, OSDI.

[38]  Mark Silberstein,et al.  Eleos: ExitLess OS Services for SGX Enclaves , 2017, EuroSys.

[39]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[40]  Miguel Correia,et al.  Spin One's Wheels? Byzantine Fault Tolerance with a Spinning Primary , 2009, 2009 28th IEEE International Symposium on Reliable Distributed Systems.