Large LDPC Codes for Big Data Storage

Current distributed storage systems mainly rely on data replication to ensure certain level of data availability and reliability. But in scenarios, like data archiving, replication is not cost effective and does not provides a robust solution to prevent data loss. A recent trend is to introduce erasure codes into the distributed storage. Inspired by the RAID system, early attempts have been focused on designing Reed-Solomon (R-S) based solutions with small block sizes. This paper investigates in details about repair traffic to apply Low Density Parity Check (LDPC) codes with relatively large block sizes. It has been demonstrated that the LDPC codes have unique advantages over R-S based solutions including low repair traffic for multiple erasures and parity erasures. The LDPC-based method is integrated with the Hadoop system with various configurations. Both theoretical analysis and simulations show that significant improvement in reliability can be achieved through using large LDPC codes without increasing the repair latency and network traffic especially for multiple erasures. Simulations also show great improvement in terms repairing latencies compared with Reed-Solomon codes. The latency is further improved through parallelism by engaging map-reduce processes from Hadoop.