Can Applications Recover from fsync Failures?

We analyze how file systems and modern data-intensive applications react to fsync failures. First, we characterize how three Linux file systems (ext4, XFS, Btrfs) behave in the presence of failures. We find commonalities across file systems (pages are always marked clean, certain block writes always lead to unavailability), as well as differences (page content and failure reporting is varied). Next, we study how five widely used applications (PostgreSQL, LMDB, LevelDB, SQLite, Redis) handle fsync failures. Our findings show that although applications use many failure-handling strategies, none are sufficient: fsync failures can cause catastrophic outcomes such as data loss and corruption. Our findings have strong implications for the design of file systems and applications that intend to provide strong durability guarantees.

[1]  Andrea C. Arpaci-Dusseau,et al.  All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications , 2014, OSDI.

[2]  Andrea C. Arpaci-Dusseau,et al.  Model-based failure analysis of journaling file systems , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[3]  Wei Hu,et al.  Scalability in the XFS File System , 1996, USENIX Annual Technical Conference.

[4]  Andrea C. Arpaci-Dusseau,et al.  Application Crash Consistency and Performance with CCFS , 2017, USENIX Annual Technical Conference.

[5]  Andrea C. Arpaci-Dusseau,et al.  Parity Lost and Parity Regained , 2008, FAST.

[6]  Robert B. Hagmann,et al.  Reimplementing the Cedar file system using logging and group commit , 1987, SOSP '87.

[7]  Margo I. Seltzer,et al.  Disk Scheduling Revisited , 1990 .

[8]  Andrea C. Arpaci-Dusseau,et al.  Analyzing the effects of disk-pointer corruption , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[9]  Christian Forfang,et al.  Evaluation of High Performance Key-Value Stores , 2014 .

[10]  Bianca Schroeder,et al.  Understanding latent sector errors and how to protect against them , 2010, TOS.

[11]  Avantika Mathur,et al.  Ext4: The Next Generation of the Ext3 File System , 2007, login Usenix Mag..

[12]  Andrea C. Arpaci-Dusseau,et al.  End-to-end Data Integrity for File Systems: A ZFS Case Study , 2010, FAST.

[13]  Hannu H. Kari.  Latent Sector Faults and Reliability of Disk Arrays , 2005 .

[14]  Andrea C. Arpaci-Dusseau,et al.  EIO: Error Handling is Occasionally Correct , 2008, FAST.

[15]  Josef Bacik,et al.  BTRFS: The Linux B-Tree Filesystem , 2013, TOS.

[16]  Bianca Schroeder,et al.  Evaluating File System Reliability on Solid State Drives , 2019, USENIX Annual Technical Conference.

[17]  Yale N. Patt,et al.  Metadata update performance in file systems , 1994, OSDI '94.

[18]  Andrea C. Arpaci-Dusseau,et al.  Consistency without ordering , 2012, FAST.

[19]  Jaemin Jung,et al.  Barrier-Enabled IO Stack for Flash Storage , 2018, FAST.

[20]  Pandian Raju,et al.  Finding Crash-Consistency Bugs with Bounded Black-Box Crash Testing , 2018, OSDI.

[21]  Steven Swanson,et al.  A study of application performance with non-volatile main memory , 2015, 2015 31st Symposium on Mass Storage Systems and Technologies (MSST).

[22]  Andrea C. Arpaci-Dusseau,et al.  IRON file systems , 2005, SOSP '05.

[23]  Remzi H. Arpaci-Dusseau.  Operating Systems: Three Easy Pieces , 2015, login Usenix Mag..

[24]  Jeffrey C. Mogul,et al.  A Better Update Policy , 1994, USENIX Summer.

[25]  Kuang-Ching Wang,et al.  The Design and Operation of CloudLab , 2019, USENIX ATC.

[26]  Shankar Pasupathy,et al.  An analysis of latent sector errors in disk drives , 2007, SIGMETRICS '07.

[27]  Andrea C. Arpaci-Dusseau,et al.  Optimistic crash consistency , 2013, SOSP.

[28]  Yong Chen,et al.  PFault: A General Framework for Analyzing the Reliability of High-Performance Parallel File Systems , 2018, ICS.

[29]  Andrea C. Arpaci-Dusseau,et al.  Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to Single Errors and Corruptions , 2017, FAST.

[30]  Andrea C. Arpaci-Dusseau,et al.  Improving file system reliability with I/O shepherding , 2007, SOSP.

[31]  Andrea C. Arpaci-Dusseau,et al.  An analysis of data corruption in the storage stack , 2008, TOS.

[32]  Chuck Silvers,et al.  UBC: An Efficient Unified I/O and Memory Caching Subsystem for NetBSD , 2000, USENIX Annual Technical Conference, FREENIX Track.