XOR's, lower bounds and MDS codes for storage

MDS erasure codes are ubiquitous in storage systems that must tolerate failures. While classic Reed-Solomon codes can provide a general-purpose MDS code for any situation, systems that require high performance rely on special-purpose codes that employ the bitwise exclusive-or (XOR) operation, and may be expressed in terms of a binary generator matrix. There are known lower bounds on the density of the generator matrix; however the number of XOR operations required to encode the generator matrix, while related to the density of the matrix, has not been explored. This paper presents a stunning and counter-intuitive result — that the encoding of k message symbols in an MDS code may require fewer than k-1 XOR operations per coding symbol. The result brings into question the lower bounds on the performance of encoding and decoding MDS erasure codes.

[1]  Liu Jing,et al.  Generalizing RDP Codes Using the Combinatorial Method , 2008, 2008 Seventh IEEE International Symposium on Network Computing and Applications.

[2]  Gang Wang,et al.  Generalizing RDP Codes Using the Combinatorial Method , 2008, NCA.

[3]  Lihao Xu,et al.  Optimizing Cauchy Reed-Solomon Codes for Fault-Tolerant Network Storage Applications , 2006, Fifth IEEE International Symposium on Network Computing and Applications (NCA'06).

[4]  Garth A. Gibson,et al.  RAID: high-performance, reliable secondary storage , 1994, CSUR.

[5]  Michael G. Pecht,et al.  Enhanced Reliability Modeling of RAID Storage Systems , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).

[6]  Mario Blaum,et al.  On Lowest Density MDS Codes , 1999, IEEE Trans. Inf. Theory.

[7]  Peter F. Corbett,et al.  Row-Diagonal Parity for Double Disk Failure Correction (Awarded Best Paper!) , 2004, USENIX Conference on File and Storage Technologies.

[8]  Jehoshua Bruck,et al.  EVENODD: An Efficient Scheme for Tolerating Double Disk Failures in RAID Architectures , 1995, IEEE Trans. Computers.

[9]  Catherine D. Schuman.  An Exploration of Optimization Algorithms and Heuristics for the Creation of Encoding and Decoding Schedules in Erasure Coding , 2011 .

[10]  Alexander Vardy,et al.  MDS array codes with independent parity symbols , 1995, Proceedings of 1995 IEEE International Symposium on Information Theory.

[11]  James Lee Hafner,et al.  Matrix methods for lost data reconstruction in erasure codes , 2005, FAST'05.

[12]  Marek Karpinski,et al.  An XOR-based erasure-resilient coding scheme , 1995 .

[13]  Catherine D. Schuman,et al.  Heuristics for optimizing matrix-based erasure codes for fault-tolerant storage systems , 2012, IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012).

[14]  Catherine D. Schuman,et al.  A Performance Evaluation and Examination of Open-Source Erasure Coding Libraries for Storage , 2009, FAST.

[15]  James S. Plank,et al.  AONT-RS: Blending Security and Performance in Dispersed Storage Systems , 2011, FAST.

[16]  Minghua Chen,et al.  On Optimizing XOR-Based Codes for Fault-Tolerant Storage Applications , 2007, 2007 IEEE Information Theory Workshop.

[17]  Mario Blaum.  A Family of MDS Array Codes with Minimal Number of Encoding Operations , 2006, 2006 IEEE International Symposium on Information Theory.

[18]  Cheng Huang,et al.  Decoding STAR code for tolerating simultaneous disk failure and silent errors , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).