Storage Support for Data-Intensive Applications on Large Scale High-Performance Computing Systems

Many believe that the state-of-the-art yet decades old high-performance computing (HPC) storage would not meet the I/O requirement of the emerging exascale mainly due to the segregation of compute and storage resources. Indeed, our simulation predicts, quantitatively, that the efficiency and availability would go towards zero as the system scales approach exascale. This work proposes a new architecture with nodelocal persistent storage. Although collocating compute and storage has been widely leveraged in cloud computing, such a system never exists in HPC. We implement a system prototype, called FusionFS, with two major design principles: maximal metadata concurrency and optimal file write, both of which are crucial to HPC applications. We also discuss FusionFSs other integral features such as hybrid and cooperative caching, efficient data access to compressed files, space-economic data redundancy, lightweight provenance tracking, and integration with data management systems.