Semantics, Specification, and Bounded Verification of Concurrent Libraries in Replicated Systems

Geo-replicated systems provide a number of desirable properties such as globally low latency, high availability, scalability, and built-in fault tolerance. Unfortunately, programming correct applications on top of such systems has proven to be very challenging, in large part because of the weak consistency guarantees they offer. These complexities are exacerbated when we try to adapt existing highly-performant concurrent libraries developed for shared-memory environments to this setting. The use of these libraries, developed with performance and scalability in mind, is highly desirable. But, identifying a suitable notion of correctness to check their validity under a weakly consistent execution model has not been well-studied, in large part because it is problematic to naïvely transplant criteria such as linearizability that has a useful interpretation in a shared-memory context to a distributed one where the cost of imposing a (logical) global ordering on all actions is prohibitive. In this paper, we tackle these issues by proposing appropriate semantics and specifications for highly-concurrent libraries in a weakly-consistent, replicated setting. We use these specifications to develop a static analysis framework that can automatically detect correctness violations of library implementations parameterized with respect to the different consistency policies provided by the underlying system. We use our framework to analyze the behavior of a number of highly non-trivial library implementations of stacks, queues, and exchangers. Our results provide the first demonstration that automated correctness checking of concurrent libraries in a weakly geo-replicated setting is both feasible and practical.

[1]  Hongseok Yang,et al.  'Cause I'm strong enough: Reasoning about consistency choices in distributed systems , 2016, POPL.

[2]  Ali Ghodsi,et al.  Highly Available Transactions: Virtues and Limitations , 2013, Proc. VLDB Endow..

[3]  Peter Müller,et al.  Static serializability analysis for causal consistency , 2018, PLDI.

[4]  Constantin Enea,et al.  Weak-consistency specification via visibility relaxation , 2019, Proc. ACM Program. Lang..

[5]  Suresh Jagannathan,et al.  Safe replication through bounded concurrency verification , 2018, Proc. ACM Program. Lang..

[6]  Constantin Enea,et al.  Replication-aware linearizability , 2019, PLDI.

[7]  Ali Ghodsi,et al.  The potential dangers of causal consistency and an explicit solution , 2012, SoCC '12.

[8]  Constantin Enea,et al.  Checking Robustness Against Snapshot Isolation , 2019, CAV.

[9]  Suresh Jagannathan,et al.  Automated Parameterized Verification of CRDTs , 2019, CAV.

[10]  Constantin Enea,et al.  Monitoring refinement via symbolic reasoning , 2015, PLDI.

[11]  Ali Ghodsi,et al.  Eventual consistency today: limitations, extensions, and beyond , 2013, CACM.

[12]  Maged M. Michael,et al.  Simple, fast, and practical non-blocking and blocking concurrent queue algorithms , 1996, PODC '96.

[13]  Sebastian Burckhardt,et al.  Principles of Eventual Consistency , 2014, Found. Trends Program. Lang..

[14]  David Holmes,et al.  Java Concurrency in Practice , 2006 .

[15]  Constantin Enea,et al.  Robustness Against Transactional Causal Consistency , 2019, CONCUR.

[16]  Rachid Guerraoui,et al.  On verifying causal consistency , 2016, POPL.

[17]  Peter Van Roy,et al.  Saturn: a Distributed Metadata Service for Causal Consistency , 2017, EuroSys.

[18]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[19]  Marvin Theimer,et al.  Session guarantees for weakly consistent replicated data , 1994, Proceedings of 3rd International Conference on Parallel and Distributed Information Systems.

[20]  Ori Lahav,et al.  On library correctness under weak memory consistency: specifying and verifying concurrent libraries under declarative consistency models , 2019, Proc. ACM Program. Lang..

[21]  Lorenzo Alvisi,et al.  I Can't Believe It's Not Causal! Scalable Causal Consistency with No Slowdown Cascades , 2017, NSDI.

[22]  Michael J. Freedman,et al.  Don't settle for eventual: scalable causal consistency for wide-area storage with COPS , 2011, SOSP.

[23]  D. M. Hutton,et al.  The Art of Multiprocessor Programming , 2008 .

[24]  Annette Bieniusa,et al.  Formal Specification and Verification of CRDTs , 2014, FORTE.

[25]  John Derrick,et al.  Making Linearizability Compositional for Partially Ordered Executions , 2018, IFM.

[26]  Sebastian Burckhardt,et al.  Replicated data types: specification, verification, optimality , 2014, POPL.

[27]  Marc Shapiro,et al.  A comprehensive study of Convergent and Commutative Replicated Data Types , 2011 .

[28]  Nir Shavit.  Data structures in the multicore age , 2011, CACM.

[29]  Mohsen Lesani,et al.  Hamsaz: replication coordination analysis and synthesis , 2019, Proc. ACM Program. Lang..

[30]  Marc Shapiro,et al.  Conflict-Free Replicated Data Types , 2011, SSS.

[31]  Rachid Guerraoui,et al.  Causal Consistency and Latency Optimality: Friend or Foe? , 2018, Proc. VLDB Endow..

[32]  Suresh Jagannathan,et al.  CLOTHO: directed test generation for weakly consistent database systems , 2019, Proc. ACM Program. Lang..

[33]  Alastair R. Beresford,et al.  Verifying strong eventual consistency in distributed systems , 2017, Proc. ACM Program. Lang..

[34]  Nir Shavit,et al.  A scalable lock-free stack algorithm , 2004, SPAA '04.

[35]  Ali Ghodsi,et al.  Coordination Avoidance in Database Systems , 2014, Proc. VLDB Endow..

[36]  Alexey Gotsman,et al.  A Framework for Transactional Consistency Models with Atomic Visibility , 2015, CONCUR.

[37]  Alexey Gotsman,et al.  Robustness against Consistency Models with Atomic Visibility , 2016, CONCUR.

[38]  Suresh Jagannathan,et al.  Declarative programming over eventually consistent data stores , 2015, PLDI.

[39]  Suresh Jagannathan,et al.  Automated Detection of Serializability Violations under Weak Consistency , 2018, CONCUR.