H5hut: A high-performance I/O library for particle-based simulations

Particle-based simulations running on large highperformance computing systems over many time steps can generate an enormous amount of particle- and field-based data for post-processing and analysis. Achieving high-performance I/O for this data, effectively managing it on disk, and interfacing it with analysis and visualization tools can be challenging, especially for domain scientists who do not have I/O and data management expertise. We present the H5hut library, an implementation of several data models for particle-based simulations that encapsulates the complexity of HDF5 and is simple to use, yet does not compromise performance.

[1]  A. Adelmann,et al.  Beam dynamics in high intensity cyclotrons including neighboring bunch effects: Model, implementation, and application , 2010, 1003.0326.

[2]  John Shalf,et al.  HDF5-FastQuery: Accelerating Complex Queries on HDF Datasets using Fast Bitmap Indices , 2005, 18th International Conference on Scientific and Statistical Database Management (SSDBM'06).

[3]  Jeffrey S. Vetter,et al.  Performance characterization and optimization of parallel I/O on the Cray XT , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[4]  Prabhat,et al.  Title High performance multivariate visual data exploration for extremely large data Permalink , 2008 .

[5]  Rajeev Thakur,et al.  Optimizing noncontiguous accesses in MPI-IO , 2002, Parallel Comput..

[6]  John Shalf,et al.  Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[7]  Jianwei Li,et al.  Parallel netCDF: A High-Performance Scientific I/O Interface , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[8]  Michael S. Warren,et al.  Robustness of Cosmological Simulations. I. Large-Scale Structure , 2004, astro-ph/0411795.

[9]  Werner Benger,et al.  Using Geometric Algebra for Navigation in Riemannian and Hard Disc Space , 2009 .

[10]  Peter Arbenz,et al.  A fast parallel Poisson solver on irregular domains applied to beam dynamics simulations , 2009, J. Comput. Phys..

[11]  Arie Shoshani,et al.  Optimizing bitmap indices with efficient compression , 2006, TODS.

[12]  Karsten Schwan,et al.  Adaptable, metadata rich IO methods for portable high performance IO , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[13]  John Bent,et al.  PLFS: a checkpoint filesystem for parallel applications , 2009, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis.