Massively Parallel I/O for Partitioned Solver Systems

This paper investigates I/O approaches for massively parallel partitioned solver systems. Typically, such systems have synchronized "loops" and write data in a well defined block I/O format consisting of a header and data portion. Our target use for such a parallel I/O subsystem is checkpoint-restart where writing is by far the most common operation and reading typically only happens during either initialization or during a restart operation because of a system failure. We compare four parallel I/O strategies: POSIX File Per Processor (1PFPP), "Poor-Man's" Parallel I/O (PMPIO), a synchronized parallel I/O (syncIO), and a "reduced blocking" strategy (rbIO). Performance tests executed on the Blue Gene/P at Argonne National Laboratory using real CFD solver data from PHASTA (an unstructured grid finite element Navier-Stokes solver) show that the syncIO strategy can achieve a read bandwidth of 47.4 GB/sec and a write bandwidth of 27.5 GB/sec using 128K processors. The "reduced-blocking" rbIO strategy achieves an actual writing performance of 17.8 GB/sec and the perceived writing performance is 166 TB/sec on Blue Gene/P using 128K processors.