Server-directed collective I/O in Panda

We present the architecture and implementation results for Panda 2.0, a library for input and output of multidimensional arrays on parallel and sequential platforms. Panda achieves remarkable performance levels on the IBM SP2, showing excellent scalability as data size increases and as the number of nodes increases, and provides throughputs close to the full capacity of the AIX file system on the SP2 we used. We argue that this good performance can be traced to Panda's use of server-directed i/o (a logical-level version of disk-directed i/o [Kotz94b]) to perform array i/o using sequential disk reads and writes, a very high level interface for collective i/o requests, and built-in facilities for arbitrary rearrangements of arrays during i/o. Other advantages of Panda's approach are ease of use, easy application portability, and a reliance on commodity system software.

[1]  David Kotz.  Expanding the potential for disk-directed I/O , 1995, Proceedings.Seventh IEEE Symposium on Parallel and Distributed Processing.

[2]  Joel H. Saltz,et al.  Jovian: a framework for optimizing parallel I/O , 1994, Proceedings Scalable Parallel Libraries Conference.

[3]  George C. Polyzos,et al.  Dynamic I/O characterization of I/O intensive scientific applications , 1994, Proceedings of Supercomputing '94.

[4]  Erik Seligman,et al.  High-Level Fault Tolerance in Distributed Programs , 1994 .

[5]  James C. French,et al.  Extensible File Systems (ELFS): An Object-Oriented Approach to High Performance File I/O , 1994, OOPSLA.

[6]  Marianne Winslett,et al.  Physical schemas for large multidimensional arrays in scientific computing applications , 1994, Seventh International Working Conference on Scientific and Statistical Database Management.

[7]  David Kotz,et al.  Disk-directed I/O for MIMD multiprocessors , 1994, OSDI '94.

[8]  David Kotz.  Throughput of Existing Multiprocessor File Systems (An Informal Study) , 1993 .

[9]  Marianne Winslett,et al.  An efficient abstract interface for multidimensional array I/O , 1994, Proceedings of Supercomputing '94.

[10]  A. Choudhary,et al.  Design and Evaluation of primitives for Parallel I/O , 1993, Supercomputing '93.

[11]  Alok N. Choudhary,et al.  The design of VIP-FS: a virtual, parallel file system for high performance parallel and distributed computing , 1995, OPSR.

[12]  David Kotz,et al.  Dynamic file-access characteristics of a production parallel scientific workload , 1994, Proceedings of Supercomputing '94.

[13]  Peter Brezany,et al.  Language, compiler and parallel database support for I/O intensive applications , 1995, HPCN Europe.

[14]  Carla Schlatter Ellis,et al.  Characterizing parallel file-access patterns on a large-scale multiprocessor , 1995, IPPS.

[15]  David Kotz.  Disk-directed I/O for an out-of-core computation , 1995, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing.

[16]  W. Gropp,et al.  Applications-driven parallel I/O , 1993, Supercomputing '93.