Decoupling lock-free data structures from memory reclamation for static analysis

Verification of concurrent data structures is one of the most challenging tasks in software verification. The topic has received considerable attention over the course of the last decade. Nevertheless, human-driven techniques remain cumbersome and notoriously difficult while automated approaches suffer from limited applicability. The main obstacle for automation is the complexity of concurrent data structures. This is particularly true in the absence of garbage collection. The intricacy of lock-free memory management paired with the complexity of concurrent data structures makes automated verification prohibitive. In this work we present a method for verifying concurrent data structures and their memory management separately. We suggest two simpler verification tasks that imply the correctness of the data structure. The first task establishes an over-approximation of the reclamation behavior of the memory management. The second task exploits this over-approximation to verify the data structure without the need to consider the implementation of the memory management itself. To make the resulting verification tasks tractable for automated techniques, we establish a second result. We show that a verification tool needs to consider only executions where a single memory location is reused. We implemented our approach and were able to verify linearizability of Michael&Scott's queue and the DGLM queue for both hazard pointers and epoch-based reclamation. To the best of our knowledge, we are the first to verify such implementations fully automatically.

[1]  Peter W. O'Hearn,et al.  Modular verification of a non-blocking stack , 2007, POPL '07.

[2]  Erez Petrank,et al.  Efficient Memory Management for Lock-Free Data Structures with Optimistic Access , 2015, SPAA.

[3]  Xinyu Feng,et al.  A rely-guarantee-based simulation for verifying concurrent program transformations , 2012, POPL '12.

[4]  Pedro Ramalhete,et al.  Brief Announcement: Hazard Eras - Non-Blocking Memory Reclamation , 2017, SPAA.

[5]  Constantin Enea,et al.  Proving Linearizability Using Forward Simulations , 2017, CAV.

[6]  Cliff B. Jones,et al.  Tentative steps toward a development method for interfering programs , 1983, TOPL.

[7]  Maurice Herlihy,et al.  Fast non-intrusive memory reclamation for highly-concurrent data structures , 2016, ISMM.

[8]  Dan Alistarh,et al.  ThreadScan: Automatic and Scalable Memory Reclamation , 2015, SPAA.

[9]  John Derrick,et al.  How to Prove Algorithms Linearisable , 2012, CAV.

[10]  Thomas A. Henzinger,et al.  Aspect-Oriented Linearizability Proofs , 2013, CONCUR.

[11]  Maged M. Michael,et al.  Simple, fast, and practical non-blocking and blocking concurrent queue algorithms , 1996, PODC '96.

[12]  Konstantinos Sagonas,et al.  Stateless model checking of the Linux kernel's hierarchical read-copy-update (tree RCU) , 2017, SPIN.

[13]  Yu Zhang,et al.  Reasoning about Optimistic Concurrency Using a Program Logic for History , 2010, CONCUR.

[14]  Joost-Pieter Katoen,et al.  Verifying Concurrent Stacks by Divergence-Sensitive Bisimulation , 2017, ArXiv.

[15]  Eran Yahav,et al.  Deriving linearizable fine-grained concurrent objects , 2008, PLDI '08.

[16]  Viktor Vafeiadis,et al.  RGSep Action Inference , 2010, VMCAI.

[17]  Mark Moir,et al.  Formal Verification of a Lazy Concurrent List-Based Set Algorithm , 2006, CAV.

[18]  Sharad Malik,et al.  Model Checking Unbounded Concurrent Lists , 2013, SPIN.

[19]  Mark Moir,et al.  Lock-free reference counting , 2002, PODC '01.

[20]  Lukás Holík,et al.  Pointer Race Freedom , 2016, VMCAI.

[21]  Maged M. Michael Safe memory reclamation for dynamic lock-free objects using atomic reads and writes , 2002, PODC '02.

[22]  Maurice Herlihy,et al.  On the power of hardware transactional memory to simplify memory management , 2011, PODC '11.

[23]  Daniel Kroening,et al.  Faster Linearizability Checking via P-Compositionality , 2015, FORTE.

[24]  FengXinyu,et al.  A rely-guarantee-based simulation for verifying concurrent program transformations , 2012 .

[25]  Omer Subasi,et al.  Simplifying Linearizability Proofs with Reduction and Abstraction , 2010, TACAS.

[26]  Ilya Sergey,et al.  Concurrent Data Structures Linked in Time , 2017, ECOOP.

[27]  Thomas A. Henzinger,et al.  Aspect-Oriented Linearizability Proofs , 2013, CONCUR.

[28]  Constantin Enea,et al.  Monitoring refinement via symbolic reasoning , 2015, PLDI.

[29]  John Derrick,et al.  Mechanically verified proof obligations for linearizability , 2011, TOPL.

[30]  Marina Papatriantafilou,et al.  Efficient and Reliable Lock-Free Memory Reclamation Based on Reference Counting , 2009, IEEE Transactions on Parallel and Distributed Systems.

[31]  Roman Manevich,et al.  Thread Quantification for Concurrent Shape Analysis , 2008, CAV.

[32]  Swarat Chaudhuri,et al.  Model Checking of Linearizability of Concurrent List Implementations , 2010, CAV.

[33]  Joseph Tassarotti,et al.  Verifying read-copy-update in a logic for weak memory , 2015, PLDI.

[34]  Wojciech M. Golab,et al.  Making objects writable , 2014, PODC '14.

[35]  John Derrick,et al.  Verifying linearizability: A comparative survey , 2014, ArXiv.

[36]  Daniel Kroening,et al.  Verification of tree-based hierarchical read-copy update in the Linux kernel , 2016, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[37]  Robert Colvin,et al.  Verifying Concurrent Data Structures by Simulation , 2005, Electron. Notes Theor. Comput. Sci..

[38]  Eran Yahav,et al.  Verifying linearizability with hindsight , 2010, PODC.

[39]  Mark Moir,et al.  DCAS is not a silver bullet for nonblocking algorithm design , 2004, SPAA '04.

[40]  Wolfgang Reif,et al.  Proving linearizability with temporal logic , 2009, Formal Aspects of Computing.

[41]  Eran Yahav,et al.  Experience with Model Checking Linearizability , 2009, SPIN.

[42]  Maged M. Michael,et al.  Correction of a Memory Management Method for Lock-Free Data Structures , 1995 .

[43]  Daniel Kroening,et al.  Partial Orders for Efficient Bounded Model Checking of Concurrent Software , 2013, CAV.

[44]  Suresh Jagannathan,et al.  Poling: SMT Aided Linearizability Proofs , 2015, CAV.

[45]  Dan Alistarh,et al.  StackTrack: an automated transactional approach to concurrent memory reclamation , 2014, EuroSys '14.

[46]  Mark Moir,et al.  Nonblocking Algorithms and Backward Simulation , 2009, DISC.

[47]  Rachid Guerraoui,et al.  Fast and Robust Memory Reclamation for Concurrent Data Structures , 2016, SPAA.

[48]  Yanhong A. Liu,et al.  Verifying Linearizability via Optimized Refinement Checking , 2013, IEEE Transactions on Software Engineering.

[49]  Maurice Herlihy,et al.  Nonblocking memory management support for dynamic-sized data structures , 2005, TOCS.

[50]  Sebastian Burckhardt,et al.  Line-up: a complete and automatic linearizability checker , 2010, PLDI '10.

[51]  Lukás Holík,et al.  Effect Summaries for Thread-Modular Analysis - Sound Analysis Despite an Unsound Heuristic , 2017, SAS.

[52]  Jonathan Walpole,et al.  Exploiting deferred destruction: an analysis of read-copy-update techniques in operating system kernels , 2004 .

[53]  Dennis Shasha,et al.  Go with the flow: compositional abstractions for concurrent data structures , 2017, Proc. ACM Program. Lang..

[54]  Bengt Jonsson Using refinement calculus techniques to prove linearizability , 2012, Formal Aspects of Computing.

[55]  Jonathan Walpole,et al.  Performance of memory reclamation for lockless synchronization , 2007, J. Parallel Distributed Comput..

[56]  Yanhong A. Liu,et al.  Model Checking Linearizability via Refinement , 2009, FM.

[57]  Michael L. Scott,et al.  Interval-based memory reclamation , 2018, PPOPP.

[58]  Parosh Aziz Abdulla,et al.  Automated Verification of Linearization Policies , 2016, SAS.

[59]  Erez Petrank,et al.  Automatic memory reclamation for lock-free data structures , 2015, OOPSLA.

[60]  Tobias Wrigstad,et al.  Type-assisted automatic garbage collection for lock-free data structures , 2017, ISMM.

[61]  Paul E. McKenney,et al.  READ-COPY UPDATE: USING EXECUTION HISTORY TO SOLVE CONCURRENCY PROBLEMS , 2002 .

[62]  Trevor Alexander Brown,et al.  Reclaiming Memory for Lock-Free Data Structures: There has to be a Better Way , 2015, PODC.

[63]  Parosh Aziz Abdulla,et al.  An integrated specification and verification technique for highly concurrent data structures for highly concurrent data structures , 2017, International Journal on Software Tools for Technology Transfer.

[64]  Oleg Travkin,et al.  SPIN as a Linearizability Checker under Weak Memory Models , 2013, Haifa Verification Conference.

[65]  Erez Petrank,et al.  Drop the anchor: lightweight memory management for non-blocking data structures , 2013, SPAA.

[66]  Viktor Vafeiadis,et al.  Modular Verification of Concurrency-Aware Linearizability , 2015, DISC.

[67]  Gavin Lowe,et al.  Testing for linearizability , 2017, Concurr. Comput. Pract. Exp..

[68]  Roman Manevich,et al.  Abstract Transformers for Thread Correlation Analysis , 2009, APLAS.

[69]  Parosh Aziz Abdulla,et al.  Correction to: An integrated specification and verification technique for highly concurrent data structures , 2013, TACAS.

[70]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[71]  Mark Moir,et al.  Formal Verification of a Practical Lock-Free Queue Algorithm , 2004, FORTE.

[72]  Lindsay Groves Reasoning about Nonblocking Concurrency using Reduction , 2007, 12th IEEE International Conference on Engineering Complex Computer Systems (ICECCS 2007).

[73]  Constantin Enea,et al.  Sound, complete, and tractable linearizability monitoring for concurrent collections , 2018, Proc. ACM Program. Lang..

[74]  Shao Jie Zhang Scalable automatic linearizability checking , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[75]  Xinyu Feng,et al.  Modular verification of linearizability with non-fixed linearization points , 2013, PLDI.

[76]  Keir Fraser,et al.  Practical lock-freedom , 2003 .

[77]  Lindsay Groves,et al.  Verifying Michael and Scott's Lock-Free Queue Algorithm using Trace Reduction , 2008, CATS.

[78]  Viktor Vafeiadis Automatically Proving Linearizability , 2010, CAV.

[79]  Alexey Gotsman,et al.  Proving Linearizability Using Partial Orders , 2017, ESOP.

[80]  Hongseok Yang,et al.  Verifying Concurrent Memory Reclamation Algorithms with Grace , 2013, ESOP.

[81]  Timothy L. Harris,et al.  A Pragmatic Implementation of Non-blocking Linked-Lists , 2001, DISC.

[82]  Eran Yahav,et al.  Comparison Under Abstraction for Verifying Linearizability , 2007, CAV.

[83]  Moshe Y. Vardi Verification of Concurrent Programs: The Automata-Theoretic Framework , 1991, Ann. Pure Appl. Log..

[84]  Sharad Malik,et al.  Model checking unbounded concurrent lists , 2015, International Journal on Software Tools for Technology Transfer.

[85]  Maged M. Michael Hazard pointers: safe memory reclamation for lock-free objects , 2004, IEEE Transactions on Parallel and Distributed Systems.

[86]  Lukás Holík,et al.  Effect Summaries for Thread-Modular Analysis , 2017, ArXiv.

[87]  Xinyu Feng,et al.  Rely-Guarantee-Based Simulation for Compositional Verification of Concurrent Program Transformations , 2014, ACM Trans. Program. Lang. Syst..

[88]  Ilya Sergey,et al.  Specifying and Verifying Concurrent Algorithms with Histories and Subjectivity , 2014, ESOP.

[89]  Wolfgang Reif,et al.  Formal Verification of a Lock-Free Stack with Hazard Pointers , 2011, ICTAC.

[90]  Ilya Sergey,et al.  Mechanized verification of fine-grained concurrent programs , 2015, PLDI.

[91]  Christoph M. Kirsch,et al.  A Scalable, Correct Time-Stamped Stack , 2015, POPL.

[92]  Michel Dagenais,et al.  Multi-core systems modeling for formal verification of parallel algorithms , 2013, OPSR.