Silver: A Scalable, Distributed, Multi-versioning, Always Growing (Ag) File System

The storage needs of users have shifted from just needing to store data to requiring a rich interface which enables the efficient query of versions, snapshots and creation of clones. Providing these features in a distributed file system while maintaining scalability, strong consistency and performance remains a challenge. In this paper we introduce Silver, a file system which leverages the Corfu distributed logging system to not only store data, but to provide fast strongly consistent snapshots, clones and multi-versioning while preserving the scalability and performance of the distributed shared log. We describe and implement Silver using a FUSE prototype and show its performance characteristics.

[1]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[2]  Tao Zou,et al.  Tango: distributed data structures over a shared log , 2013, SOSP.

[3]  John H. Hartman,et al.  Zebra: A Striped Network File System , 1992 .

[4]  Ittai Abraham,et al.  Replex: A Scalable, Highly Available Multi-Index Data Store , 2016, USENIX Annual Technical Conference.

[5]  Andrea C. Arpaci-Dusseau,et al.  De-indirection for flash-based SSDs with nameless writes , 2012, FAST.

[6]  Dahlia Malkhi,et al.  Beyond block I/O: implementing a distributed shared log in hardware , 2013, SYSTOR '13.

[7]  Robert S. Fabry,et al.  A fast file system for UNIX , 1984, TOCS.

[8]  Andrea C. Arpaci-Dusseau,et al.  End-to-end Data Integrity for File Systems: A ZFS Case Study , 2010, FAST.

[9]  R. S. Fabry,et al.  A fast file system for UNIX , 1984, TOCS.

[10]  Carlos Maltzahn,et al.  Ceph: a scalable, high-performance distributed file system , 2006, OSDI '06.

[11]  Dahlia Malkhi,et al.  CORFU: A distributed shared log , 2013, TOCS.

[12]  John H. Hartman,et al.  The Zebra striped network file system , 1995, TOCS.

[13]  Arkady Kanevsky,et al.  FlexVol: Flexible, Efficient File Volume Virtualization in WAFL , 2008, USENIX Annual Technical Conference.

[14]  Sean Quinlan,et al.  A cached WORM file system , 1991, Softw. Pract. Exp..

[15]  Josef Bacik,et al.  BTRFS: The Linux B-Tree Filesystem , 2013, TOS.

[16]  Daniel J. Abadi,et al.  CalvinFS: Consistent WAN Replication and Scalable Metadata Management for Distributed File Systems , 2015, FAST.

[17]  Koji Sato,et al.  The Linux implementation of a log-structured file system , 2006, OPSR.