Using Crash Hoare logic for certifying the FSCQ file system

FSCQ is the first file system with a machine-checkable proof (using the Coq proof assistant) that its implementation meets its specification and whose specification includes crashes. FSCQ provably avoids bugs that have plagued previous file systems, such as performing disk writes without sufficient barriers or forgetting to zero out directory blocks. If a crash happens at an inopportune time, these bugs can lead to data loss. FSCQ's theorems prove that, under any sequence of crashes followed by reboots, FSCQ will recover the file system correctly without losing data. To state FSCQ's theorems, this paper introduces the Crash Hoare logic (CHL), which extends traditional Hoare logic with a crash condition, a recovery procedure, and logical address spaces for specifying disk states at different abstraction levels. CHL also reduces the proof effort for developers through proof automation. Using CHL, we developed, specified, and proved the correctness of the FSCQ file system. Although FSCQ's design is relatively simple, experiments with FSCQ running as a user-level file system show that it is sufficient to run Unix applications with usable performance. FSCQ's specifications and proofs required significantly more work than the implementation, but the work was manageable even for a small team of a few researchers.

[1]  Adam Wright,et al.  Local Reasoning for the POSIX File System , 2014, ESOP.

[2]  Sidney Amani,et al.  File systems deserve verification too! , 2013, PLOS '13.

[3]  Danfeng Zhang,et al.  Ironclad Apps: End-to-End Security via Automated Full-System Verification , 2014, OSDI.

[4]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[5]  Lauretta O. Osho,et al.  Axiomatic Basis for Computer Programming , 2013 .

[6]  Adam Chlipala,et al.  The bedrock structured programming system: combining generative metaprogramming and hoare logic in an extensible program verifier , 2013, ICFP.

[7]  Andrea C. Arpaci-Dusseau,et al.  All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications , 2014, OSDI.

[8]  Austin T. Clements,et al.  The scalable commutativity rule: designing scalable software for multicore processors , 2013, SOSP.

[9]  Martin C. Rinard,et al.  Verifying quantitative reliability for programs that execute on unreliable hardware , 2013, OOPSLA.

[10]  Martin C. Rinard,et al.  Automatic detection and repair of errors in data structures , 2003, OOPSLA '03.

[11]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[12]  Xi Wang,et al.  Verdi: a framework for implementing and formally verifying distributed systems , 2015, PLDI.

[13]  June Andronick.  Formally Proved Anti-tearing Properties of Embedded C Code , 2006, Second International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (isola 2006).

[14]  Andrea C. Arpaci-Dusseau,et al.  A Study of Linux File System Evolution , 2013, FAST.

[15]  Gerard J. Holzmann,et al.  A mini challenge: build a verifiable filesystem , 2007, Formal Aspects of Computing.

[16]  Andrea C. Arpaci-Dusseau,et al.  Beyond Storage APIs: Provable Semantics for Storage Stacks , 2015, HotOS.

[17]  Roxana Geambasu,et al.  Experiences with formal specification of fault-tolerant file systems , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[18]  Andrea C. Arpaci-Dusseau,et al.  SQCK: A Declarative File System Checker , 2008, OSDI.

[19]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[20]  David Walker,et al.  Static typing for a faulty lambda calculus , 2006, ICFP '06.

[21]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[22]  Gidon Ernst,et al.  Inside a Verified Flash File System: Transactions and Garbage Collection , 2015, VSTTE.

[23]  Remzi H. Arpaci-Dusseau.  Operating Systems: Three Easy Pieces , 2015, login Usenix Mag..

[24]  John C. Reynolds,et al.  Separation logic: a logic for shared mutable data structures , 2002, Proceedings 17th Annual IEEE Symposium on Logic in Computer Science.

[25]  Daniel Jackson,et al.  Formal Modeling and Analysis of a Flash Filesystem in Alloy , 2008, ABZ.

[26]  Junfeng Yang,et al.  EXPLODE: a lightweight, general system for finding serious storage system errors , 2006, OSDI '06.

[27]  Viktor Kuncak,et al.  Verifying a File System Implementation , 2004, ICFEM.

[28]  Luke D. Postema,et al.  The Institute of Electrical and Electronics Engineers , 1963, Nature.

[29]  José Nuno Oliveira,et al.  An Integrated Formal Methods Tool-Chain and Its Application to Verifying a File System Model , 2009, SBMF.

[30]  G. Ernst,et al.  Universität Augsburg Crash-Safe Refinement for a Verified Flash File System , 2014 .

[31]  Jim Woodcock,et al.  POSIX and the Verification Grand Challenge: A Roadmap , 2008, 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008).

[32]  David Walker,et al.  Fault-tolerant typed assembly language , 2007, PLDI '07.

[33]  Eddie Kohler,et al.  Specifying Crash Safety for Storage Systems , 2015, HotOS.

[34]  Wim H. Hesselink,et al.  Formalizing a hierarchical file system , 2009, Formal Aspects of Computing.

[35]  Chris Hawblitzel,et al.  Safe to the last instruction: automated verification of a type-safe operating system , 2011, CACM.

[36]  Yu Guo,et al.  Deep Specifications and Certified Abstraction Layers , 2015, POPL.

[37]  Gidon Ernst,et al.  Development of a Verified Flash File System , 2014, ABZ.

[38]  Mark Lillibridge,et al.  Torturing Databases for Fun and Profit , 2014, OSDI.

[39]  Xavier Leroy,et al.  A Formally Verified Compiler Back-end , 2009, Journal of Automated Reasoning.

[40]  Philippa Gardner,et al.  Fault-Tolerant Resource Reasoning , 2015, APLAS.

[41]  Gidon Ernst,et al.  Verification of a Virtual Filesystem Switch , 2013, VSTTE.

[42]  Adam Chlipala,et al.  Mostly-automated verification of low-level programs in computational separation logic , 2011, PLDI '11.

[43]  Hongseok Yang,et al.  Views: compositional reasoning for concurrent programs , 2013, POPL.

[44]  Michael Norrish,et al.  seL4: formal verification of an OS kernel , 2009, SOSP '09.

[45]  TU MarkusWenzel.  Some aspects of Unix file-system security , 2001 .

[46]  Hamid Pirahesh,et al.  ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging , 1998 .

[47]  Xi Wang,et al.  Jitk: A Trustworthy In-Kernel Interpreter Infrastructure , 2014, OSDI.

[48]  Junfeng Yang,et al.  Automatically generating malicious disks using symbolic execution , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[49]  Stephen C. Tweedie,et al.  Journaling the Linux ext2fs Filesystem , 2008 .

[50]  Leslie Lamport,et al.  The temporal logic of actions , 1994, TOPL.