aHDFS: An Erasure-Coded Data Archival System for Hadoop Clusters

In this paper, we propose an erasure-coded data archival system called <italic>aHDFS</italic> for Hadoop clusters, where <inline-formula><tex-math notation="LaTeX">$RS(k+r,k)$</tex-math><alternatives> <inline-graphic xlink:href="chen-ieq1-2706686.gif"/></alternatives></inline-formula> codes are employed to archive data replicas in the Hadoop distributed file system or HDFS. We develop two archival strategies (i.e., <italic> aHDFS-Grouping</italic> and <italic>aHDFS-Pipeline</italic>) in <italic>aHDFS</italic> to speed up the data archival process. <italic>aHDFS-Grouping</italic> - a MapReduce-based data archiving scheme - keeps each mapper’s intermediate output <italic>Key-Value</italic> pairs in a local key-value store. With the local store in place, <italic>aHDFS-Grouping</italic> merges all the intermediate <italic>key-value</italic> pairs with the same key into one single <italic>key-value</italic> pair, followed by shuffling the single <italic>Key-Value</italic> pair to reducers to generate final parity blocks. <italic>aHDFS-Pipeline</italic> forms a data archival pipeline using multiple data node in a Hadoop cluster. <italic>aHDFS-Pipeline</italic> delivers the merged single <italic>key-value</italic> pair to a subsequent node’s local key-value store. Last node in the pipeline is responsible for outputting parity blocks. We implement <italic>aHDFS</italic> in a real-world Hadoop cluster. The experimental results show that <italic> aHDFS-Grouping</italic> and <italic>aHDFS-Pipeline</italic> speed up <italic>Baseline</italic>’s shuffle and reduce phases by a factor of 10 and 5, respectively. When block size is larger than 32 MB, <italic>aHDFS</italic> improves the performance of <italic>HDFS-RAID</italic> and <italic>HDFS-EC</italic> by approximately 31.8 and 15.7 percent, respectively.