Capacity bounds for distributed storage

One of the primary objectives of a distributed storage system is to reliably store large amounts of source data for long durations using a large number $N$ of unreliable storage nodes, each with $c$ bits of storage capacity. Storage nodes fail randomly over time and are replaced with nodes of equal capacity initialized to zeroes, and thus bits are erased at some rate $E$. To maintain recoverability of the source data, a repairer continually reads data over a network from nodes at a rate $R$, and generates and writes data to nodes based on the read data. The distributed storage source data capacity is the maximum amount of source data that can be reliably stored for long periods of time. We prove the distributed storage source data capacity asymptotically approaches $\left(1-\frac{E}{2 \cdot R}\right) \cdot N \cdot c$ as $N$ and $R$ grow. This equation expresses a fundamental trade-off between network traffic and storage overhead to reliably store source data.

[1]  Darrell D. E. Long,et al.  Protecting against rare event failures in archival systems , 2009, 2009 IEEE International Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication Systems.

[2]  John Kubiatowicz,et al.  Erasure Coding Vs. Replication: A Quantitative Comparison , 2002, IPTPS.

[3]  Cory Hill,et al.  f4: Facebook's Warm BLOB Storage System , 2014, OSDI.

[4]  Van-Anh Truong,et al.  Availability in Globally Distributed Storage Systems , 2010, OSDI.

[5]  Cheng Huang,et al.  Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads , 2012, FAST.

[6]  Dimitris S. Papailiopoulos,et al.  XORing Elephants: Novel Erasure Codes for Big Data , 2013, Proc. VLDB Endow..

[7]  Kannan Ramchandran,et al.  A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster , 2013, HotStorage.

[8]  Cheng Huang,et al.  On the Locality of Codeword Symbols , 2011, IEEE Transactions on Information Theory.

[9]  Thomas Stockhammer,et al.  RaptorQ Forward Error Correction Scheme for Object Delivery , 2011, RFC.

[10]  Andrew V. Goldberg,et al.  A prototype implementation of archival Intermemory , 1999, DL '99.

[11]  Shubhangi Saraf,et al.  Maximally Recoverable Codes for Grid-like Topologies , 2016, SODA.

[12]  Andreas Haeberlen,et al.  Proactive Replication for Data Durability , 2006, IPTPS.

[13]  Andreas Haeberlen,et al.  Glacier: highly durable, decentralized storage despite massive correlated failures , 2005, NSDI.

[14]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[15]  Alexandros G. Dimakis,et al.  Network Coding for Distributed Storage Systems , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[16]  Ben Y. Zhao,et al.  OceanStore: an architecture for global-scale persistent storage , 2000, SIGP.

[17]  Luigi Rizzo,et al.  Effective erasure codes for reliable computer communication protocols , 1997, CCRV.

[18]  P. Vijay Kumar,et al.  An Explicit, Coupled-Layer Construction of a High-Rate MSR Code with Low Sub-Packetization Level, Small Field Size and All-Node Repair , 2016, ArXiv.

[19]  Shahram Yousefi,et al.  Locality-aware fountain codes for massive distributed storage systems , 2015, 2015 IEEE 14th Canadian Workshop on Information Theory (CWIT).

[20]  Andreas Haeberlen,et al.  Efficient Replica Maintenance for Distributed Storage Systems , 2006, NSDI.

[21]  Marek Karpinski,et al.  An XOR-based erasure-resilient coding scheme , 1995 .

[22]  Emina Soljanin,et al.  Raptor codes based distributed storage algorithms for wireless sensor networks , 2008, 2008 IEEE International Symposium on Information Theory.

[23]  S. Janson Tail bounds for sums of geometric and exponential variables , 2017, 1709.08157.

[24]  Alexandros G. Dimakis,et al.  Repairable Fountain Codes , 2014, IEEE J. Sel. Areas Commun..

[25]  Vinod M. Prabhakaran,et al.  Distributed Fountain Codes for Networked Storage , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[26]  Cheng Huang,et al.  Erasure Coding in Windows Azure Storage , 2012, USENIX Annual Technical Conference.

[27]  Lakshmi Ganesh,et al.  Lazy Means Smart: Reducing Repair Bandwidth Costs in Erasure-coded Distributed Storage , 2014, SYSTOR 2014.