Efficient Metadata Management in Large Distributed File Systems

Metadata management in large distributed systems is a critical aspect of overall system performance. Directory subtree partitioning and pure hashing are two common techniques used for metadata management. Neither of these techniques can support large systems with high concurrent accesses due to the conflicting need for both load balancing and directory update efficiency. We present a flexible approach that addresses the problems of the existing techniques. The approach, called Lazy Hybrid Hashed Hierarchical directory management, takes advantage of the load balancing and other performance benefits of hashing, while maintaining a hierarchical directory structure and metadata logging to support lazy metadata relocation and updates. A dual-entry Access Control List provides direct permission calculation to the target file, avoiding the overhead of traversing the pathname for permission check.