Bidl: A High-throughput, Low-latency Permissioned Blockchain Framework for Datacenter Networks

A permissioned blockchain framework typically runs an efficient Byzantine consensus protocol and is attractive to deploy fast trading applications among a large number of mutually untrusted participants (e.g., companies). Unfortunately, all existing permissioned blockchain frameworks adopt sequential workflows for invoking the consensus protocol and executing applications' transactions, making the performance of these applications much lower than deploying them in traditional systems (e.g., in-datacenter stock exchange). We propose Bidl, the first permissioned blockchain framework highly optimized for datacenter networks. We leverage the network ordering in such networks to create a shepherded parallel workflow, which carries a sequencer to parallelize the consensus protocol and transaction execution speculatively. However, the presence of malicious participants (e.g., a malicious sequencer) can easily perturb the parallel workflow to greatly degrade Bidl's performance. To achieve stable high performance, Bidl efficiently shepherds all participants by detecting their misbehaviors, and performs denylist-based view changes to replace or deny malicious participants. Compared with three fast permissioned blockchain frameworks, Bidl's parallel workflow reduces applications' latency by up to 72.7% and improves their throughput by up to 4.3x in the presence of malicious participants. Bidl is suitable to be integrated with traditional stock exchange systems. Bidl's code is released on github.com/hku-systems/bidl.

[1]  Alex C. Snoeren,et al.  Passive Realtime Datacenter Fault Detection and Localization , 2017, NSDI.

[2]  Jianbo Gao,et al.  Performance Analysis of the Libra Blockchain: An Experimental Study , 2019, 2019 2nd International Conference on Hot Information-Centric Networking (HotICN).

[3]  Daniel Davis Wood,et al.  ETHEREUM: A SECURE DECENTRALISED GENERALISED TRANSACTION LEDGER , 2014 .

[4]  Sun Jun,et al.  Potential Risks of Hyperledger Fabric Smart Contracts , 2019, 2019 IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE).

[5]  Onur Mutlu,et al.  A Large Scale Study of Data Center Network Reliability , 2018, Internet Measurement Conference.

[6]  Marko Vukolic,et al.  The Next 700 BFT Protocols , 2015, ACM Trans. Comput. Syst..

[7]  T. S. Eugene Ng,et al.  The Impact of Virtualization on Network Performance of Amazon EC2 Data Center , 2010, 2010 Proceedings IEEE INFOCOM.

[8]  Maurice Herlihy,et al.  An Empirical Study of Speculative Concurrency in Ethereum Smart Contracts , 2019, Tokenomics.

[9]  Silvio Micali,et al.  Algorand: Scaling Byzantine Agreements for Cryptocurrencies , 2017, IACR Cryptol. ePrint Arch..

[10]  Ramakrishna Kotla,et al.  High throughput Byzantine fault tolerance , 2004, International Conference on Dependable Systems and Networks, 2004.

[11]  Zhetao Li,et al.  Consortium Blockchain for Secure Energy Trading in Industrial Internet of Things , 2018, IEEE Transactions on Industrial Informatics.

[12]  Michael Dahlin,et al.  Making Byzantine Fault Tolerant Systems Tolerate Byzantine Faults , 2009, NSDI.

[13]  Ali Ghodsi,et al.  Highly Available Transactions: Virtues and Limitations , 2013, Proc. VLDB Endow..

[14]  Moein Sabounchi,et al.  Towards resilient networked microgrids: Blockchain-enabled peer-to-peer electricity trading mechanism , 2017, 2017 IEEE Conference on Energy Internet and Energy System Integration (EI2).

[15]  Xiaozhou Li,et al.  NetChain: Scale-Free Sub-RTT Coordination , 2018, NSDI.

[16]  Alysson Neves Bessani,et al.  From Byzantine Consensus to BFT State Machine Replication: A Latency-Optimal Transformation , 2012, 2012 Ninth European Dependable Computing Conference.

[17]  Mariana Raykova,et al.  RapidChain: Scaling Blockchain via Full Sharding , 2018, CCS.

[18]  Mohammad Hamdaqa,et al.  Blockchain-Based E-Voting System , 2018, 2018 IEEE 11th International Conference on Cloud Computing (CLOUD).

[19]  Ittai Abraham,et al.  HotStuff: BFT Consensus in the Lens of Blockchain , 2018, 1803.05069.

[20]  Marcos K. Aguilera,et al.  Failure detection and consensus in the crash-recovery model , 2000, Distributed Computing.

[21]  Robert P. Bartlett,et al.  How Rigged Are Stock Markets? Evidence from Microsecond Timestamps , 2016 .

[22]  Stephen Lee,et al.  FastFabric: Scaling Hyperledger Fabric to 20,000 Transactions per Second , 2019, 2019 IEEE International Conference on Blockchain and Cryptocurrency (ICBC).

[23]  Benny Pinkas,et al.  SBFT: a Scalable Decentralized Trust Infrastructure for Blockchains , 2018, ArXiv.

[24]  Jens Dittrich,et al.  Blurring the Lines between Blockchains and Database Systems: the Case of Hyperledger Fabric , 2019, SIGMOD Conference.

[25]  Andrew Brook Evolution and Practice: Low-latency Distributed Applications in Finance , 2015, ACM Queue.

[26]  Philipp Jovanovic,et al.  OmniLedger: A Secure, Scale-Out, Decentralized Ledger via Sharding , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[27]  Marko Vukolic,et al.  Hyperledger fabric: a distributed operating system for permissioned blockchains , 2018, EuroSys.

[28]  Jialin Li,et al.  Eris: Coordination-Free Consistent Transactions Using In-Network Concurrency Control , 2017, SOSP.

[29]  Vitalik Buterin A NEXT GENERATION SMART CONTRACT & DECENTRALIZED APPLICATION PLATFORM , 2015 .

[30]  Arati Baliga,et al.  Performance Evaluation of the Quorum Blockchain Platform , 2018, ArXiv.

[31]  Keke Gai,et al.  Blockchain-enabled Data Provenance in Cloud Datacenter Reengineering , 2019, BSCI.

[32]  John Lane,et al.  Byzantine replication under attack , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[33]  Leslie Lamport,et al.  Paxos Made Simple , 2001 .

[34]  Jeffrey S. Chase,et al.  Strong accountability for network storage , 2007, TOS.

[35]  Ittay Eyal,et al.  Ostraka: Secure Blockchain Scaling by Node Sharding , 2019, 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW).

[36]  Rüdiger Kapitza,et al.  Towards Low-Latency Byzantine Agreement Protocols Using RDMA , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W).

[37]  Budi Rahardjo,et al.  Blockchain based e-voting recording system design , 2017, 2017 11th International Conference on Telecommunication Systems Services and Applications (TSSA).

[38]  Yang Wang,et al.  All about Eve: Execute-Verify Replication for Multi-Core Servers , 2012, OSDI.

[39]  Hao Wang,et al.  Monoxide: Scale out Blockchains with Asynchronous Consensus Zones , 2019, NSDI.

[40]  Song Guo,et al.  Byzantine-Resilient Secure Software-Defined Networks with Multiple Controllers in Cloud , 2014, IEEE Transactions on Cloud Computing.

[41]  Ramakrishna Kotla,et al.  Zyzzyva , 2007, SOSP.

[42]  Xin Jin,et al.  Pegasus: Tolerating Skewed Workloads in Distributed Storage with In-Network Coherence Directories , 2020, OSDI.

[43]  Zuqing Zhu,et al.  On Fast and Coordinated Data Backup in Geo-Distributed Optical Inter-Datacenter Networks , 2015, Journal of Lightwave Technology.

[44]  Yi Ge,et al.  Design and Analysis of a Distributed Multi-leg Stock Trading System , 2011, 2011 31st International Conference on Distributed Computing Systems.

[45]  C. Moallemi,et al.  The Cost of Latency ∗ , 2009 .

[46]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.

[47]  Kishor S. Trivedi,et al.  Performance Modeling of Hyperledger Fabric (Permissioned Blockchain Network) , 2018, 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA).

[48]  Beng Chin Ooi,et al.  A Transactional Perspective on Execute-order-validate Blockchains , 2020, SIGMOD Conference.

[49]  Andreas Haeberlen,et al.  PeerReview: practical accountability for distributed systems , 2007, SOSP.

[50]  Joel Hasbrouck,et al.  Low-latency trading $ , 2013 .

[51]  Alysson Neves Bessani,et al.  State Machine Replication for the Masses with BFT-SMART , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[52]  S. Nakamoto,et al.  Bitcoin: A Peer-to-Peer Electronic Cash System , 2008 .

[53]  John K. Ousterhout,et al.  In Search of an Understandable Consensus Algorithm , 2014, USENIX ATC.

[54]  Jialin Li,et al.  Just Say NO to Paxos Overhead: Replacing Consensus with Network Ordering , 2016, OSDI.

[55]  Ahmed Ben Ayed,et al.  A Conceptual Secure Blockchain Based Electronic Voting System , 2017 .

[56]  Fahad R. Dogar,et al.  Measuring and Improving the Reliability of Wide-Area Cloud Paths , 2017, WWW.

[57]  Fernando Pedone,et al.  NetPaxos: consensus at network speed , 2015, SOSR.

[58]  Hua Chen,et al.  Pingmesh: A Large-Scale System for Data Center Network Latency Measurement and Analysis , 2015, SIGCOMM.

[59]  Alysson Bessani,et al.  A Byzantine Fault-Tolerant Ordering Service for the Hyperledger Fabric Blockchain Platform , 2017, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[60]  Andreas Haeberlen,et al.  The Synchronous Data Center , 2019, HotOS.

[61]  Ittai Abraham,et al.  HotStuff: BFT Consensus with Linearity and Responsiveness , 2019, PODC.

[62]  Jia Liu,et al.  Toward Low-Cost and Stable Blockchain Networks , 2020, ICC 2020 - 2020 IEEE International Conference on Communications (ICC).

[63]  Vivien Quéma,et al.  RBFT: Redundant Byzantine Fault Tolerance , 2013, 2013 IEEE 33rd International Conference on Distributed Computing Systems.

[64]  Sachin Shetty,et al.  ProvChain: A Blockchain-Based Data Provenance Architecture in Cloud Environment with Enhanced Privacy and Availability , 2017, 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID).

[65]  Yibo Zhu,et al.  Congestion Control for Cross-Datacenter Networks , 2019, 2019 IEEE 27th International Conference on Network Protocols (ICNP).

[66]  Mor Harchol-Balter,et al.  Reducing Latency via Redundant Requests: Exact Analysis , 2015, SIGMETRICS 2015.

[67]  Miguel Correia,et al.  Spin One's Wheels? Byzantine Fault Tolerance with a Spinning Primary , 2009, 2009 28th IEEE International Symposium on Reliable Distributed Systems.

[68]  Leonid Barenboim,et al.  Dynamic SDN Controller Load Balancing , 2019, Future Internet.

[69]  Miguel Oom Temudo de Castro,et al.  Practical Byzantine fault tolerance , 1999, OSDI '99.

[70]  Jialin Li,et al.  Designing Distributed Systems Using Approximate Synchrony in Data Center Networks , 2015, NSDI.

[71]  David Broman,et al.  Approximate Synchrony: An Abstraction for Distributed Almost-Synchronous Systems , 2015, CAV.

[72]  Dahlia Malkhi,et al.  CORFU: A Shared Log Design for Flash Clusters , 2012, NSDI.