Using Object Based Files for High Performance Parallel I/O

We contend that the scalable I/O problem in high performance computing is largely due to the legacy view of a file as a linear sequence of bytes. In this paper we introduce an alternative to the traditional "flat file " that uses the information contained in file views to partition a file into an optimal set of objects, minimizing locking contention and simplifying the lock management strategy. We illustrate the use of an object based cache added to ROMIO to efficiently and transparently add object-based file capabilities to MPI-IO. We analyze the performance of our system using the FLASH-IO benchmark, and demonstrate a substantial performance improvement over the standard ROMIO implementation.

[1]  Alex Rapaport,et al.  Mpi-2: extensions to the message-passing interface , 1997 .

[2]  Wei-keng Liao,et al.  DAChe: Direct Access Cache System for Parallel I/O , 2005 .

[3]  Rajeev Thakur,et al.  Users guide for ROMIO: A high-performance, portable MPI-IO implementation , 1997 .

[4]  Wei-keng Liao,et al.  Noncontiguous access through MPI-IO , 2003 .

[5]  Marianne Winslett,et al.  Faster collective output through active buffering , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[6]  Frank B. Schmuck,et al.  GPFS: A Shared-Disk File System for Large Computing Clusters , 2002, FAST.

[7]  Rajeev Thakur,et al.  An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays , 1996, Sci. Program..

[8]  Rajeev Thakur,et al.  Optimizing noncontiguous accesses in MPI-IO , 2002, Parallel Comput..

[9]  B. Fryxell,et al.  FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes , 2000 .

[10]  George K. Thiruvathukal,et al.  Scalable Approaches for Supporting MPI-IO Atomicity , 2006, Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06).

[11]  E. Lusk,et al.  An abstract-device interface for implementing portable parallel-I/O interfaces , 1996, Proceedings of 6th Symposium on the Frontiers of Massively Parallel Computation (Frontiers '96).

[12]  Robert Latham,et al.  The Impact of File Systems on MPI-IO Scalability , 2004, PVM/MPI.

[13]  Robert B. Ross,et al.  Noncontiguous I/O accesses through MPI-IO , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..