Improving I/O Throughput of Scientific Applications Using Transparent Parallel Compression

Increasing number of cores in parallel computer systems are allowing scientific simulations to be executed with increasing spatial and temporal granularity. However, this also implies that increasing larger-sized datasets need to be output, stored, managed, and then visualized and/or analyzed using a variety of methods. In examining the possibility of using compression to accelerate all of these steps, we focus on two important questions: "Can compression help save time when data is output from, or input into, a parallel program?", and "How can a scientist's effort in using compression with a parallel program be minimized?". We focus on Pnet CDF, and show how transparent compression can be supported, thus allowing an existing simulation program to start outputting and storing data in a compressed fashion, and similarly, allow a data analysis application to read compressed data. We address challenges in supporting compression when parallel writes are being performed. In our experiments, we first analyze the effects of using compression with micro benchmarks, and then, continue our evaluation using a scientific simulation application, and two data analysis applications. While we obtain up to a factor of 2 improvement in performance for micro benchmarks, the execution time of simulation application is improved up to 22%, and the maximum speedup of data analysis applications is 1.83(with an average speedup of 1.36).

[1]  Bogdan Nicolae,et al.  High Throughput Data-Compression for Cloud Storage , 2010, Globe.

[2]  Martin Burtscher,et al.  FPC: A High-Speed Compressor for Double-Precision Floating-Point Data , 2009, IEEE Transactions on Computers.

[3]  G.G. Langdon,et al.  Data compression , 1988, IEEE Potentials.

[4]  Yi Wang,et al.  SDQuery DSI: Integrating data management support with a wide area data transfer protocol , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[5]  Martin Isenburg,et al.  Fast and Efficient Compression of Floating-Point Data , 2006, IEEE Transactions on Visualization and Computer Graphics.

[6]  Jianwei Li,et al.  Parallel netCDF: A High-Performance Scientific I/O Interface , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[7]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[8]  Robert Latham,et al.  ISOBAR Preconditioner for Effective and High-throughput Lossless Data Compression , 2012, 2012 IEEE 28th International Conference on Data Engineering.

[9]  Jian Yin,et al.  Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[10]  Bronis R. de Supinski,et al.  McrEngine: a scalable checkpointing system using data-aware aggregation and compression , 2012, HiPC 2012.

[11]  Jeremy Iverson,et al.  Fast and Effective Lossy Compression Algorithms for Scientific Datasets , 2012, Euro-Par.

[12]  Yi Wang,et al.  Supporting a Light-Weight Data Management Layer over HDF5 , 2013, 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing.

[13]  A. Srinivasan,et al.  High throughput compression of floating point numbers on graphical processing units , 2012, 2012 2nd IEEE International Conference on Parallel, Distributed and Grid Computing.

[14]  Robert B. Ross,et al.  ISOBAR hybrid compression-I/O interleaving for large-scale parallel I/O optimization , 2012, HPDC '12.

[15]  Fred Douglis On the role of compression in distributed systems , 1992, EW 5.

[16]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[17]  Karsten Schwan,et al.  Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) , 2008, CLADE '08.