Position: short object lifetimes require a delete-optimized storage system

Early file systems were designed with the expectation that data would typically be read from disk many times before being deleted; on-disk structures were therefore optimized for reading. As main memory sizes increased, more read requests could be satisfied from data cached in memory, motivating file system designs that optimize write performance. Here, we describe how one might build a storage system that optimizes not only reading and writing, but creation and deletion as well. Efficiency is achieved, in part, by automating deletion based on relative retention values rather than requiring data be deleted explicitly by an application. This approach is well suited to an emerging class of applications that process data at consistently high rates of ingest. This paper explores trade-offs in clustering data by retention value and age and examines the effects of allowing the retention values to change under application control.

[1]  Larry J. Stockmeyer,et al.  An Age-Threshold Algorithm for Garbage Collection in Log-Structured Arrays and File Systems , 1998 .

[2]  Jim Zelenka,et al.  Informed prefetching and caching , 1995, SOSP.

[3]  Olivia R. Liu Sheng Dynamic file migration in distributed computer systems , 1990, CACM.

[4]  Alfons Kemper,et al.  Efficient bulk deletes in relational databases , 2001, Proceedings 17th International Conference on Data Engineering.

[5]  R. S. Fabry,et al.  A fast file system for UNIX , 1984, TOCS.

[6]  Frederick Reiss,et al.  TelegraphCQ: Continuous Dataflow Processing for an Uncertain World , 2003, CIDR.

[7]  Jun Wang,et al.  WOLF - A Novel Reordering Write Buffer to Boost the Performance of Log-Structured File Systems , 2002, FAST.

[8]  Peter Druschel,et al.  Anticipatory scheduling: a disk scheduling framework to overcome deceptive idleness in synchronous I/O , 2001, SOSP.

[9]  RosenblumMendel,et al.  The design and implementation of a log-structured file system , 1991 .

[10]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[11]  Michael Stonebraker,et al.  The Aurora and Medusa Projects , 2003, IEEE Data Eng. Bull..

[12]  Carl Staelin,et al.  An Implementation of a Log-Structured File System for UNIX , 1993, USENIX Winter.

[13]  Margo I. Seltzer,et al.  Heuristic Cleaning Algorithms in Log-Structured File Systems , 1995, USENIX.

[14]  Fred Douglis,et al.  Beating the I/O bottleneck: a case for log-structured file systems , 1989, OPSR.

[15]  Norman C. Hutchinson,et al.  Deciding when to forget in the Elephant file system , 1999, SOSP.