Using concurrent relational logic with helpers for verifying the AtomFS file system

Concurrent file systems are pervasive but hard to correctly implement and formally verify due to nondeterministic interleavings. This paper presents AtomFS, the first formally-verified, fine-grained, concurrent file system, which provides linearizable interfaces to applications. The standard way to prove linearizability requires modeling linearization point of each operation---the moment when its effect becomes visible atomically to other threads. We observe that path inter-dependency, where one operation (like rename) breaks the path integrity of other operations, makes the linearization point external and thus poses a significant challenge to prove linearizability. To overcome the above challenge, this paper presents Concurrent Relational Logic with Helpers (CRL-H), a framework for building verified concurrent file systems. CRL-H is made powerful through two key contributions: (1) extending prior approaches using fixed linearization points with a helper mechanism where one operation of the thread can logically help other threads linearize their operations; (2) combining relational specifications and rely/guarantee conditions for relational and compositional reasoning. We have successfully applied CRL-H to verify the linearizability of AtomFS directly in C code. All the proofs are mechanized in Coq. Evaluations show that AtomFS speeds up file system workloads by utilizing fine-grained, multicore concurrency.

[1]  Xinyu Feng,et al.  Modular verification of linearizability with non-fixed linearization points , 2013, PLDI 2013.

[2]  John Derrick,et al.  Verifying Linearisability with Potential Linearisation Points , 2011, FM.

[3]  Lars Birkedal,et al.  ReLoC: A Mechanised Relational Logic for Fine-Grained Concurrency , 2018, LICS.

[4]  Zhong Shao,et al.  Certified concurrent abstraction layers , 2018, PLDI.

[5]  Lauretta O. Osho,et al.  Axiomatic Basis for Computer Programming , 2013 .

[6]  Xi Wang,et al.  Verdi: a framework for implementing and formally verifying distributed systems , 2015, PLDI.

[7]  Wei Hu,et al.  Scalability in the XFS File System , 1996, USENIX Annual Technical Conference.

[8]  Viktor Vafeiadis,et al.  A Marriage of Rely/Guarantee and Separation Logic , 2007, CONCUR.

[9]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[10]  Thomas A. Henzinger,et al.  Automatic Linearizability Proofs of Concurrent Objects with Cooperating Updates , 2013, CAV.

[11]  R. Card,et al.  Design and Implementation of the Second Extended Filesystem , 2001 .

[12]  Xinyu Feng,et al.  A Practical Verification Framework for Preemptive OS Kernels , 2016, CAV.

[13]  Kai Engelhardt,et al.  Data Refinement: Model-Oriented Proof Methods and their Comparison , 1998 .

[14]  Butler W. Lampson,et al.  Verifying concurrent software using movers in CSPEC , 2018, OSDI.

[15]  Stephen Brookes.  A semantics for concurrent separation logic , 2007, Theor. Comput. Sci..

[16]  M. Frans Kaashoek,et al.  Scaling a file system to many cores using an operation log , 2017, SOSP.

[17]  Xinyu Feng,et al.  On the Relationship Between Concurrent Separation Logic and Assume-Guarantee Reasoning , 2007, ESOP.

[18]  Serdar Tasiran,et al.  Automated and Modular Refinement Reasoning for Concurrent Programs , 2015, CAV.

[19]  Zhong Shao,et al.  Safety and Liveness of MCS Lock - Layer by Layer , 2017, APLAS.

[20]  Gian Ntzik,et al.  Reasoning about POSIX file systems , 2016 .

[21]  Viktor Vafeiadis,et al.  Modular fine-grained concurrency verification , 2008 .

[22]  Peter W. O'Hearn,et al.  Local Reasoning about Programs that Alter Data Structures , 2001, CSL.

[23]  Nancy A. Lynch,et al.  Forward and Backward Simulations: I. Untimed Systems , 1995, Inf. Comput..

[24]  John Derrick,et al.  Mechanically verified proof obligations for linearizability , 2011, TOPL.

[25]  Adam Chlipala,et al.  Using Crash Hoare logic for certifying the FSCQ file system , 2015, USENIX Annual Technical Conference.

[26]  Sidney Amani,et al.  Cogent: Verifying High-Assurance File System Implementations , 2016, ASPLOS.

[27]  Willem-Paul de Roever,et al.  Data Refinement by Willem-Paul de Roever , 1998 .

[28]  Joseph Tassarotti,et al.  Verifying concurrent, crash-safe systems with Perennial , 2019, SOSP.

[29]  Cliff B. Jones,et al.  Tentative steps toward a development method for interfering programs , 1983, TOPL.

[30]  Adam Chlipala,et al.  Chapar: certified causally consistent distributed key-value stores , 2016, POPL.

[31]  Ben C. Moszkowski,et al.  Compositional Reasoning Using Interval Temporal Logic and Tempura , 1997, COMPOS.

[32]  Jian Huang,et al.  Understanding Security Vulnerabilities in File Systems , 2019, APSys '19.

[33]  Angela Demke Brown,et al.  Breaking Apart the VFS for Managing File Systems , 2018, HotStorage.

[34]  Willem-Paul de Roever,et al.  Data Refinement: Theory , 1998 .

[35]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[36]  Xinyu Feng,et al.  Compositional verification of termination-preserving refinement of concurrent programs , 2014, CSL-LICS.

[37]  Zhong Shao,et al.  Toward Compositional Verification of Interruptible OS Kernels and Device Drivers , 2016, Journal of Automated Reasoning.

[38]  Tej Chajed,et al.  Verifying an I/O-concurrent file system , 2017 .

[39]  Adam Chlipala,et al.  Proving confidentiality in a file system using DiskSec , 2018, OSDI.

[40]  Nicolas Christin,et al.  Push-Button Verification of File Systems via Crash Refinement , 2016, USENIX Annual Technical Conference.

[41]  John C. Reynolds,et al.  Separation logic: a logic for shared mutable data structures , 2002, Proceedings 17th Annual IEEE Symposium on Logic in Computer Science.

[42]  Alexey Gotsman,et al.  Proving Linearizability Using Partial Orders , 2017, ESOP.

[43]  John Derrick,et al.  Verifying Linearisability , 2015, ACM Comput. Surv..

[44]  Xinyu Feng.  Local rely-guarantee reasoning , 2009, POPL '09.

[45]  Peter W. O'Hearn,et al.  Abstraction for concurrent objects , 2009, Theor. Comput. Sci..

[46]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[47]  Mark Moir,et al.  Formal Verification of a Lazy Concurrent List-Based Set Algorithm , 2006, CAV.

[48]  Andrea C. Arpaci-Dusseau,et al.  A Study of Linux File System Evolution , 2013, FAST.

[49]  Lars Birkedal,et al.  Logical relations for fine-grained concurrency , 2013, POPL.

[50]  Xinyu Feng,et al.  A rely-guarantee-based simulation for verifying concurrent program transformations , 2012, POPL '12.

[51]  Changwoo Min,et al.  Cross-checking semantic correctness: the case of finding file system bugs , 2015, SOSP.

[52]  Adam Chlipala,et al.  Verifying a high-performance crash-safe file system using a tree specification , 2017, SOSP.

[53]  Zhong Shao,et al.  CertiKOS: An Extensible Architecture for Building Certified Concurrent OS Kernels , 2016, OSDI.

[54]  Viktor Vafeiadis.  Concurrent Separation Logic and Operational Semantics , 2011, MFPS.

[55]  Michael Norrish,et al.  seL4: formal verification of an OS kernel , 2009, SOSP '09.

[56]  Richard J. Lipton,et al.  Reduction: a method of proving properties of parallel programs , 1975, CACM.

[57]  Nir Shavit,et al.  A scalable lock-free stack algorithm , 2010, J. Parallel Distributed Comput..

[58]  Philippa Gardner,et al.  A Concurrent Specification of POSIX File Systems , 2018, ECOOP.

[59]  N. Lynch,et al.  Forward and backward simulations , 1993 .

[60]  Jozef Hooman,et al.  Concurrency Verification: Introduction to Compositional and Noncompositional Methods , 2001, Cambridge Tracts in Theoretical Computer Science.

[61]  Mark A. Hillebrand,et al.  VCC: A Practical System for Verifying Concurrent C , 2009, TPHOLs.

[62]  Andrea C. Arpaci-Dusseau,et al.  All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications , 2014, OSDI.

[63]  Maurice Herlihy,et al.  The art of multiprocessor programming , 2020, PODC '06.