Minimum storage regenerating codes for all parameters

Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an [n, k, d]-(α) MSR code C over Fq is defined as follows. Using such a code C, a file F consisting of αk symbols over Fq can be distributed among n nodes, each storing α symbols, in such a way that: . the file F can be recovered by downloading the content of any k of the n nodes; and . the content of any failed node can be reconstructed by accessing any d of the remaining n -1 nodes and downloading α/(d-k+1) symbols from each of these nodes. A common practical requirement for regenerating codes is to have the original file F available in uncoded form on some k of the n nodes, known as systematic nodes. In this case, several authors relax the defining node-repair condition above, requiring the optimal repair bandwidth of dα/(d-k+1) symbols for systematic nodes only. We shall call such codes systematic-repair MSR codes. Unfortunately, explicit constructions of [n, k, d] MSR codes are known only for certain special cases: either low rate, namely k/n ≤ 0.5, or high repair connectivity, namely d = n -1. Although setting d = n - 1 minimizes the repair bandwidth, it may be impractical to connect to all the remaining nodes in order to repair a single failed node. Our main result in this paper is an explicit construction of systematic-repair [n, k, d] MSR codes for all possible values of parameters n, k, d. In particular, we construct systematic-repair MSR codes of high rate k/n > 0.5 and low repair connectivity k ≤ d ≤ n - 1. Such codes were not previously known to exist. In order to construct these codes, we solve simultaneously several repair scenarios, each of which is expressible as an interference alignment problem. Extension of our results beyond systematic repair remains an open problem.

[1]  Venkatesan Guruswami,et al.  Repairing Reed-Solomon Codes , 2015, IEEE Transactions on Information Theory.

[2]  Jehoshua Bruck,et al.  Long MDS codes for optimal repair bandwidth , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[3]  Jacob T. Schwartz,et al.  Fast Probabilistic Algorithms for Verification of Polynomial Identities , 1980, J. ACM.

[4]  Jehoshua Bruck,et al.  Zigzag Codes: MDS Array Codes With Optimal Rebuilding , 2011, IEEE Transactions on Information Theory.

[5]  Harald Øverby,et al.  General Sub-Packetized Access-Optimal Regenerating Codes , 2016, IEEE Communications Letters.

[6]  Venkatesan Guruswami,et al.  New MDS codes with small sub-packetization and near-optimal repair bandwidth , 2016, ArXiv.

[7]  Tetsunao Matsuta,et al.  国際会議開催報告:2013 IEEE International Symposium on Information Theory , 2013 .

[8]  Natalia Silberstein,et al.  Constructions of High-Rate MSR Codes over Small Fields , 2015, 1505.00919.

[9]  Alexandros G. Dimakis,et al.  Searching for Minimum Storage Regenerating Codes , 2009, ArXiv.

[10]  P. Vijay Kumar,et al.  An Explicit, Coupled-Layer Construction of a High-Rate MSR Code with Low Sub-Packetization Level, Small Field Size and All-Node Repair , 2016, ArXiv.

[11]  Arman Fazeli,et al.  Minimum Storage Regenerating Codes for All Parameters , 2017, IEEE Transactions on Information Theory.

[12]  A. Robert Calderbank,et al.  An Improved Sub-Packetization Bound for Minimum Storage Regenerating Codes , 2013, IEEE Transactions on Information Theory.

[13]  Sriram Vishwanath,et al.  Progress on high-rate MSR codes: Enabling arbitrary number of helper nodes , 2016, 2016 Information Theory and Applications Workshop (ITA).

[14]  Nihar B. Shah,et al.  Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction , 2010, IEEE Transactions on Information Theory.

[15]  Kannan Ramchandran,et al.  Asymptotic Interference Alignment for Optimal Repair of MDS Codes in Distributed Storage , 2013, IEEE Transactions on Information Theory.

[16]  Yunnan Wu,et al.  A Construction of Systematic MDS Codes With Minimum Repair Bandwidth , 2009, IEEE Transactions on Information Theory.

[17]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[18]  Alexandros G. Dimakis,et al.  Network Coding for Distributed Storage Systems , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[19]  Alexander Barg,et al.  Explicit Constructions of High-Rate MDS Array Codes With Optimal Repair Bandwidth , 2016, IEEE Transactions on Information Theory.

[20]  Jehoshua Bruck,et al.  MDS array codes with optimal rebuilding , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[21]  GhemawatSanjay,et al.  The Google file system , 2003 .

[22]  Cheng Huang,et al.  Polynomial length MDS codes with optimal repair in distributed storage , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[23]  P. Vijay Kumar,et al.  An alternate construction of an access-optimal regenerating code with optimal sub-packetization level , 2015, 2015 Twenty First National Conference on Communications (NCC).

[24]  Dimitris S. Papailiopoulos,et al.  Repair Optimal Erasure Codes Through Hadamard Designs , 2011, IEEE Transactions on Information Theory.

[25]  Yunnan Wu,et al.  Reducing repair traffic for erasure coding-based storage via interference alignment , 2009, 2009 IEEE International Symposium on Information Theory.

[26]  Kannan Ramchandran,et al.  Distributed Storage Codes With Repair-by-Transfer and Nonachievability of Interior Points on the Storage-Bandwidth Tradeoff , 2010, IEEE Transactions on Information Theory.

[27]  Kannan Ramchandran,et al.  Exact-repair MDS codes for distributed storage using interference alignment , 2010, 2010 IEEE International Symposium on Information Theory.

[28]  Alexander Barg,et al.  Explicit Constructions of Optimal-Access MDS Codes With Nearly Optimal Sub-Packetization , 2016, IEEE Transactions on Information Theory.

[29]  Jehoshua Bruck,et al.  On codes for optimal rebuilding access , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).