Intelligent Metadata Management for a Petabyte-scale File System

In petabyte-scale distributed file systems that decouple read and write from metadata operations, behavior of the metadata server cluster will be critical to overall system performance. We examine aspects of the workload that make it difficult to distribute effectively, and present a fe w potential strategies to demonstrate the issues involved. F inally, we describe the advantages of intelligent metadata management and a simulation environment we have developed to validate design possibilities.

[1]  Mahadev Satyanarayanan,et al.  Andrew: a distributed personal computing environment , 1986, CACM.

[2]  Andrew R. Cherenson,et al.  The Sprite network operating system , 1988, Computer.

[3]  Mahadev Satyanarayanan,et al.  Coda: A Highly Available File System for a Distributed Workstation Environment , 1990, IEEE Trans. Computers.

[4]  James Lau,et al.  File System Design for an NFS File Server Appliance , 1994, USENIX Winter.

[5]  Carl Smith,et al.  NFS Version 3: Design and Implementation , 1994, USENIX Summer.

[6]  Dror G. Feitelson,et al.  The Vesta parallel file system , 1996, TOCS.

[7]  Randy H. Katz,et al.  RAMA: An Easy-to-Use, High-Performance Parallel File System , 1997, Parallel Comput..

[8]  Michael J. Callahan,et al.  The InterMezzo File System , 1999 .

[9]  Thomas E. Anderson,et al.  A Comparison of File System Workloads , 2000, USENIX Annual Technical Conference, General Track.

[10]  Scott A. Brandt,et al.  Reliability mechanisms for very large storage systems , 2003, 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings..

[11]  GhemawatSanjay,et al.  The Google file system , 2003 .

[12]  Ethan L. Miller,et al.  Replication under scalable hashing: a family of algorithms for scalable decentralized data distribution , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[13]  E. L. Miller,et al.  Efficient Metadata Management in Large Distributed File Systems , .