A practical analysis of low-density parity-check erasure codes for wide-area storage applications

As peer-to-peer and widely distributed storage systems proliferate, the need to perform efficient erasure coding, instead of replication, is crucial to performance and efficiency. Low-density parity-check (LDPC) codes have arisen as alternatives to standard erasure codes, such as Reed-Solomon codes, trading off vastly improved decoding performance for inefficiencies in the amount of data that must be acquired to perform decoding. The scores of papers written on LDPC codes typically analyze their collective and asymptotic behavior. Unfortunately, their practical application requires the generation and analysis of individual codes for finite systems. This paper attempts to illuminate the practical considerations of LDPC codes for peer-to-peer and distributed storage systems. The three main types of LDPC codes are detailed, and a huge variety of codes are generated, then analyzed using simulation. This analysis focuses on the performance of individual codes for finite systems, and addresses several important heretofore unanswered questions about employing LDPC codes in real-world systems.

[1]  Stephen B. Wicker,et al.  Fundamentals of Codes, Graphs, and Iterative Decoding , 2002 .

[2]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[3]  Amin Shokrollahi,et al.  Codes and Graphs , 2000, STACS.

[4]  Zheng Zhang,et al.  Reperasure: Replication Protocol using Erasure-code , 2002 .

[5]  Marek Karpinski,et al.  An XOR-based erasure-resilient coding scheme , 1995 .

[6]  R. Storn,et al.  Design of efficient erasure codes with differential evolution , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[7]  Ben Y. Zhao,et al.  OceanStore: an architecture for global-scale persistent storage , 2000, SIGP.

[8]  H. Jin,et al.  Irregular repeat accumulate codes , 2000 .

[9]  James S. Plank,et al.  Improving the performance of coordinated checkpointers on networks of workstations using RAID techniques , 1996, Proceedings 15th Symposium on Reliable Distributed Systems.

[10]  Amin Shokrollahi,et al.  Raptor codes , 2011, IEEE Transactions on Information Theory.

[11]  Ben Y. Zhao,et al.  Maintenance-Free Global Data Storage , 2001, IEEE Internet Comput..

[12]  Matthew S. Allen,et al.  The Livny and Plank-Beck Problems: Studies in Data Movement on the Computational Grid , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[13]  Witold Litwin,et al.  LH*RS: a high-availability scalable distributed data structure using Reed Solomon Codes , 2000, SIGMOD '00.

[14]  Michael Mitzenmacher,et al.  Accessing multiple mirror sites in parallel: using Tornado codes to speed up downloads , 1999, IEEE INFOCOM '99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No.99CH36320).

[15]  Walter A. Burkhard,et al.  Disk array storage system reliability , 1993, FTCS-23 The Twenty-Third International Symposium on Fault-Tolerant Computing.

[16]  Giuseppe Caire,et al.  Design methods for irregular repeat-accumulate codes , 2003, IEEE Transactions on Information Theory.

[17]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[18]  Michael Mitzenmacher,et al.  Analysis of random processes via And-Or tree evaluation , 1998, SODA '98.

[19]  Emre Telatar,et al.  Finite-length analysis of low-density parity-check codes on the binary erasure channel , 2002, IEEE Trans. Inf. Theory.

[20]  J. Plank,et al.  On the Practical Use of LDPC Erasure Codes for Distributed Storage Applications , 2003 .

[21]  Daniel A. Spielman,et al.  Practical loss-resilient codes , 1997, STOC '97.

[22]  Garth A. Gibson,et al.  RAID: high-performance, reliable secondary storage , 1994, CSUR.

[23]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[24]  B. Marx.  The Visual Display of Quantitative Information , 1985 .

[25]  Michael Luby,et al.  LT codes , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[26]  Michael Luby,et al.  A digital fountain approach to reliable distribution of bulk data , 1998, SIGCOMM '98.

[27]  Robert J. McEliece.  Achieving the Shannon Limit: A Progress Report , 2000 .

[28]  John Kubiatowicz,et al.  Erasure Coding Vs. Replication: A Quantitative Comparison , 2002, IPTPS.

[29]  M. Aminshokrollahi.  New sequences of linear time erasure codes approaching the channel capacity , 1999 .