A transparent parallel I/O environment

We describe TPIE, a Transparent Parallel I/O Environment. TPIE is a system designed to bridge the gap between current theoretical knowledge about the construction of I/O-optimal algorithms on parallel disk systems and the design and implementation of parallel I/O systems. We discuss the design of TPIE and its interface, the structure of a typical implementation, applications of the system, our prototype, and future research directions. The initial goal of our work is a prototype system to demonstrate: 1) that optimal algorithms can be made to run eeciently on parallel I/O devices; and 2) that high level hardware independent interfaces to the I/O paradigms required to implement such algorithms can be provided to application programmers. The TPIE interface is designed to be portable across a variety of parallel hardware platforms; thus code that runs eeciently on one machine will run eeciently on others. Longer term goals for TPIE include extending the prototype in ways that will permit it to function as an extremely exible parallel I/O system capable of supporting a wide range of applications, such as matrix computation, computational geometry, graph manipulation, and database management.