Liquid Cloud Storage

A liquid system provides durable object storage based on spreading redundantly generated data across a network of hundreds to thousands of potentially unreliable storage nodes. A liquid system uses a combination of a large code, lazy repair, and flow storage organization. We show that a liquid system can be operated to enable flexible and essentially optimal combinations of storage durability, storage overhead, repair bandwidth usage, and access performance.

[1]  Kannan Ramchandran,et al.  A “Hitchhiker’s” Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers , 2014 .

[2]  Cheng Huang,et al.  On the Locality of Codeword Symbols , 2011, IEEE Transactions on Information Theory.

[3]  Kannan Ramchandran,et al.  Having Your Cake and Eating It Too: Jointly Optimal Erasure Codes for I/O, Storage, and Network-bandwidth , 2015, FAST.

[4]  Carlos Maltzahn,et al.  Ceph: a scalable, high-performance distributed file system , 2006, OSDI '06.

[5]  Lakshmi Ganesh,et al.  Lazy Means Smart: Reducing Repair Bandwidth Costs in Erasure-coded Distributed Storage , 2014, SYSTOR 2014.

[6]  Ju Wang,et al.  Windows Azure Storage: a highly available cloud storage service with strong consistency , 2011, SOSP.

[7]  Michael Luby.  Capacity bounds for distributed storage , 2016, ArXiv.

[8]  Rodrigo Rodrigues,et al.  High Availability in DHTs: Erasure Coding vs. Replication , 2005, IPTPS.

[9]  Sachin Katti,et al.  Copysets: Reducing the Frequency of Data Loss in Cloud Storage , 2013, USENIX Annual Technical Conference.

[10]  Vincent Roca,et al.  Reed-solomon forward error correction (FEC) schemes, RFC 5510 , 2009 .

[11]  Cheng Huang,et al.  Giza: Erasure Coding Objects across Global Data Centers , 2017, USENIX Annual Technical Conference.

[12]  Dimitris S. Papailiopoulos,et al.  XORing Elephants: Novel Erasure Codes for Big Data , 2013, Proc. VLDB Endow..

[13]  Luigi Rizzo,et al.  Effective erasure codes for reliable computer communication protocols , 1997, CCRV.

[14]  Martin Thomson,et al.  Hypertext Transfer Protocol Version 2 (HTTP/2) , 2015, RFC.

[15]  Andreas Haeberlen,et al.  Proactive Replication for Data Durability , 2006, IPTPS.

[16]  Marek Karpinski,et al.  An XOR-based erasure-resilient coding scheme , 1995 .

[17]  John Kubiatowicz,et al.  Erasure Coding Vs. Replication: A Quantitative Comparison , 2002, IPTPS.

[18]  Emina Soljanin,et al.  Coding for fast content download , 2012, 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[19]  Renato Recio,et al.  A Remote Direct Memory Access Protocol Specification , 2007, RFC.

[20]  Patrick P. C. Lee,et al.  Repair Pipelining for Erasure-Coded Storage , 2017, USENIX Annual Technical Conference.

[21]  Cory Hill,et al.  f4: Facebook's Warm BLOB Storage System , 2014, OSDI.

[22]  Van-Anh Truong,et al.  Availability in Globally Distributed Storage Systems , 2010, OSDI.

[23]  Stefan Savage,et al.  Total Recall: System Support for Automated Availability Management , 2004, NSDI.

[24]  Cheng Huang,et al.  Erasure Coding in Windows Azure Storage , 2012, USENIX Annual Technical Conference.

[25]  Yanpei Chen,et al.  Understanding TCP Incast and Its Implications for Big Data Workloads , 2012, login Usenix Mag..

[26]  Alexandros G. Dimakis,et al.  Network Coding for Distributed Storage Systems , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[27]  Thomas Stockhammer,et al.  RaptorQ Forward Error Correction Scheme for Object Delivery , 2011, RFC.