Characterizing parallel file-access patterns on a large-scale multiprocessor

High-performance parallel file systems are needed to satisfy tremendous I/O requirements of parallel scientific applications. The design of such high-performance parallel file systems depends on a comprehensive understanding of the expected workload, but so far there have been very few usage studies of multiprocessor file systems. This paper is part of the CHARISMA project, which intends to fill this void by measuring real file-system workloads on various production parallel machines. In particular, here we present results from the CM-5 at the National Center for Supercomputing Applications. Our results are unique because we collect information about nearly every individual I/O request from the mix of jobs running on the machine. Analysis of the traces leads to various recommendations for parallel file-system design. >

[1]  Rick Floyd.  Short-Term File Reference Patterns in a UNIX Environment, , 1986 .

[2]  Dror G. Feitelson,et al.  Parallel access to files in the Vesta file system , 1993, Supercomputing '93. Proceedings.

[3]  E. DeBenedictis,et al.  nCUBE parallel I/O software , 1992, Eleventh Annual International Phoenix Conference on Computers and Communication [1992 Conference Proceedings].

[4]  Michael L. Powell,et al.  The DEMOS file system , 1977, SOSP '77.

[5]  Mary Baker,et al.  Measurements of a distributed file system , 1991, SOSP '91.

[6]  David Kotz,et al.  Integrating Theory and Practice in Parallel File Systems , 1993 .

[7]  David Levine,et al.  Applications-driven parallel I/O , 1993, Supercomputing '93. Proceedings.

[8]  K. K. Ramakrishnan,et al.  Analysis of file I/O traces in commercial computing environments , 1992, SIGMETRICS '92/PERFORMANCE '92.

[9]  Thomas W. Crockett.  File concepts for parallel I/O , 1989, Proceedings of the 1989 ACM/IEEE Conference on Supercomputing (Supercomputing '89).

[10]  Prithviraj Banerjee,et al.  A study of I/O behavior of perfect benchmarks on a multiprocessor , 1990, ISCA '90.

[11]  Daniel A. Reed,et al.  File archive activity in a supercomputing environment , 1993, ICS '93.

[12]  Randy H. Katz,et al.  Input/output behavior of supercomputing applications , 1991, Proceedings of the 1991 ACM/IEEE Conference on Supercomputing (Supercomputing '91).

[13]  G. C. Polyzos,et al.  A static analysis of I/O characteristics of scientific applications in a production workload , 1993, Supercomputing '93.

[14]  John A. Kunze,et al.  A trace-driven analysis of the UNIX 4.2 BSD file system , 1985, SOSP '85.

[15]  David Kotz,et al.  Multiprocessor file system interfaces , 1993, [1993] Proceedings of the Second International Conference on Parallel and Distributed Information Systems.

[16]  David Kotz,et al.  Dynamic file-access characteristics of a production parallel scientific workload , 1994, Proceedings of Supercomputing '94.

[17]  Alok N. Choudhary,et al.  High-performance I/O for massively parallel computers: problems and prospects , 1994, Computer.

[18]  Carla Schlatter Ellis,et al.  Directory Reference Patterns in Hierarchical File Systems , 1989, IEEE Trans. Knowl. Data Eng..

[19]  Alan Jay Smith.  Analysis of Long Term File Reference Patterns for Application to File Migration Algorithms , 1981, IEEE Transactions on Software Engineering.

[20]  Michael Stumm,et al.  HFS: A Flexible File System for large-scale Multiprocessors , 1993 .

[21]  W. Gropp,et al.  Applications-driven parallel I/O , 1993, Supercomputing '93.

[22]  Carla Schlatter Ellis,et al.  Characterizing Parallel File-access Patterns on a , 1995 .

[23]  Michael L. Best,et al.  CMMD I/O: a parallel Unix I/O , 1993, [1993] Proceedings Seventh International Parallel Processing Symposium.

[24]  Peter Dibble,et al.  A parallel interleaved file system , 1990 .

[25]  P. Messina,et al.  Architectural requirements of parallel scientific applications with explicit communication , 1993, ISCA '93.

[26]  Juan M. Porcar.  File migration in distributed computer systems , 1982, Perform. Evaluation.

[27]  D. G. Feitelson,et al.  Parallel access to files in the Vesta file system , 1993, Supercomputing '93.

[28]  James C. French,et al.  Performance Measurement of the Concurrent File System of the Intel iPSC/2 Hypercube , 1993, J. Parallel Distributed Comput..

[29]  Paul J. Roy.  Unix File Access and Caching in a Multicomputer Environment , 1993, USENIX MACH Symposium.