Integrating collective I/O and cooperative caching into the "clusterfile" parallel file system

This paper presents the integration of two collective I/O techniques into the Clusterfile parallel file system: disk-directed I/O and two-phase I/O. We show that global cooperative cache management improves the collective I/O performance. The solution focuses on integrating disk parallelism with other types of parallelism: memory (by buffering and caching on several nodes), network (by parallel I/O scheduling strategies) and processors (by redistributing the I/O related computation over several nodes). The performance results show considerable throughput increases over ROMIO's extended two-phase I/O.

[1]  David Kotz,et al.  Disk-directed I/O for MIMD multiprocessors , 1994, OSDI '94.

[2]  Anna R. Karlin,et al.  Implementing cooperative prefetching and caching in a globally-managed memory system , 1998, SIGMETRICS '98/PERFORMANCE '98.

[3]  Daniel A. Reed,et al.  A Comparison of Logical and Physical Parallel I/o pAtterns , 1998, Int. J. High Perform. Comput. Appl..

[4]  Marianne Winslett,et al.  Parallel I/O performance of fine grained data distributions , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[5]  Michael Dahlin,et al.  Cooperative caching: using remote client memory to improve file system performance , 1994, OSDI '94.

[6]  Alok N. Choudhary,et al.  Improved parallel I/O via a two-phase run-time access strategy , 1993, CARN.

[7]  Walter B. Ligon,et al.  An Overview of the Parallel Virtual File System , 1999 .

[8]  Carla Schlatter Ellis,et al.  File-Access Characteristics of Parallel Scientific Workloads , 1996, IEEE Trans. Parallel Distributed Syst..

[9]  R. Bagrodia,et al.  Parallel Simulation of Parallel File Systems and I/O Programs , 1997, ACM/IEEE SC 1997 Conference (SC'97).

[10]  Andrew A. Chien,et al.  PPFS: a high performance portable parallel file system , 1995, ICS '95.

[11]  Rajesh Bordawekar.  Implementation of collective I/O in the Intel Paragon parallel file system: initial experiences , 1997, ICS '97.

[12]  H. Apte,et al.  Serverless Network File Systems , 2006 .

[13]  Marianne Winslett,et al.  Server-Directed Collective I/O in Panda , 1995, Proceedings of the IEEE/ACM SC95 Conference.

[14]  Toni Cortes,et al.  Paca: a Distributed File System Cache for Parallel Machines. Performance under Unix-like Workload , 1995 .

[15]  Message Passing Interface Forum.  MPI: A message - passing interface standard , 1994 .

[16]  Florin Isaila,et al.  Clusterfile: a flexible physical layout parallel file system , 2001, Proceedings 42nd IEEE Symposium on Foundations of Computer Science.

[17]  Rajeev Thakur,et al.  Data sieving and collective I/O in ROMIO , 1998, Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation.

[18]  Ravi Jain,et al.  Heuristics for Scheduling I/O Operations , 1997, IEEE Trans. Parallel Distributed Syst..

[19]  Michael Metcalf,et al.  High performance Fortran , 1995 .

[20]  E. DeBenedictis,et al.  nCUBE parallel I/O software , 1992, Eleventh Annual International Phoenix Conference on Computers and Communication [1992 Conference Proceedings].

[21]  Florin Isaila,et al.  View I/O: improving the performance of non-contiguous I/O , 2003, 2003 Proceedings IEEE International Conference on Cluster Computing.

[22]  Rob VanderWijngaart,et al.  NAS Parallel Benchmarks I/O Version 2.4. 2.4 , 2002 .

[23]  Frank B. Schmuck,et al.  GPFS: A Shared-Disk File System for Large Computing Clusters , 2002, FAST.

[24]  Dror G. Feitelson,et al.  The Vesta parallel file system , 1996, TOCS.

[25]  Shikharesh Majumdar,et al.  Performance of parallel I/O scheduling strategies on a network of workstations , 2001, Proceedings. Eighth International Conference on Parallel and Distributed Systems. ICPADS 2001.