VLL: a lock manager redesign for main memory database systems

Lock managers are increasingly becoming a bottleneck in database systems that use pessimistic concurrency control. In this paper, we introduce very lightweight locking (VLL), an alternative approach to pessimistic concurrency control for main memory database systems, which avoids almost all overhead associated with traditional lock manager operations. We also propose a protocol called selective contention analysis (SCA), which enables systems implementing VLL to achieve high transactional throughput under high-contention workloads. We implement these protocols both in a traditional single-machine multi-core database server setting and in a distributed database where data are partitioned across many commodity machines in a shared-nothing cluster. Furthermore, we show how VLL and SCA can be extended to enable range locking. Our experiments show that VLL dramatically reduces locking overhead and thereby increases transactional throughput in both settings.

[1]  Daniel J. Abadi,et al.  Lightweight Locking for Main Memory Database Systems , 2012, Proc. VLDB Endow..

[2]  Ippokratis Pandis,et al.  CMU-CS-10-101 1 Data-Oriented Transaction Execution , 2010 .

[3]  Jim Gray,et al.  Notes on Data Base Operating Systems , 1978, Advanced Course: Operating Systems.

[4]  Ian Rae,et al.  F1: A Distributed SQL Database That Scales , 2013, Proc. VLDB Endow..

[5]  H. T. Kung,et al.  On optimistic concurrency control , 1981 .

[6]  Eddie Kohler,et al.  Speedy transactions in multicore in-memory databases , 2013, SOSP.

[7]  Daniel J. Abadi,et al.  The case for determinism in database systems , 2010, Proc. VLDB Endow..

[8]  Harumi A. Kuno,et al.  Efficient Locking Techniques for Databases on Modern Hardware , 2012, ADMS@VLDB.

[9]  Ippokratis Pandis,et al.  Data-oriented transaction execution , 2010, Proc. VLDB Endow..

[10]  Michael Stonebraker,et al.  OLTP through the looking glass, and what we found there , 2008, SIGMOD Conference.

[11]  J. T. Robinson,et al.  On optimistic methods for concurrency control , 1979, TODS.

[12]  Viktor Leis,et al.  Processing in the Hybrid OLTP & OLAP Main-Memory Database System HyPer , 2013, IEEE Data Eng. Bull..

[13]  Michael Stonebraker,et al.  Concurrency Control and Consistency of Multiple Copies of Data in Distributed Ingres , 1979, IEEE Transactions on Software Engineering.

[14]  Tobin J. Lehman,et al.  Locking and Latching in a Memory-Resident Database System , 1992, VLDB.

[15]  Irving L. Traiger,et al.  Granularity of Locks and Degrees of Consistency in a Shared Data Base , 1998, IFIP Working Conference on Modelling in Data Base Management Systems.

[16]  Andreas Reuter,et al.  Transaction Processing: Concepts and Techniques , 1992 .

[17]  Heon Young Yeom,et al.  A scalable lock manager for multicores , 2013, SIGMOD '13.

[18]  Miron Livny,et al.  Concurrency control performance modeling: alternatives and implications , 1987, TODS.

[19]  Bruce G. Lindsay,et al.  Transaction management in the R* distributed database management system , 1986, TODS.

[20]  Divyakant Agrawal,et al.  MaaT: Effective and scalable coordination of distributed transactions in the cloud , 2014, Proc. VLDB Endow..

[21]  Craig Freedman,et al.  Hekaton: SQL server's memory-optimized OLTP engine , 2013, SIGMOD '13.

[22]  Christopher Frost,et al.  Spanner: Google's Globally-Distributed Database , 2012, OSDI.

[23]  C. Mohan,et al.  Recovery and Coherency-Control Protocols for Fast Intersystem Page Transfer and Fine-Granularity Locking in a Shared Disks Transaction Environment , 1991, VLDB.

[24]  Babak Falsafi,et al.  Shore-MT: a scalable storage manager for the multicore era , 2009, EDBT '09.

[25]  Tobin Jon Lehman.  Design and performance evaluation of a main memory relational database system (t tree) , 1986 .

[26]  Alexander Thomasian,et al.  Two-phase locking performance and its thrashing behavior , 1993, TODS.

[27]  Sándor Dominich,et al.  an advanced course , 2005 .

[28]  Ashok M. Joshi,et al.  Adaptive Locking Strategies in a Multi-node Data Sharing Environment , 1991, VLDB.

[29]  Tobin J. Lehman,et al.  The Design and Performance Evaluation of a Lock Manager for a Memory-Resident Database System , 1996, Performance of Concurrency Control Mechanisms in Centralized Database Systems.

[30]  David B. Lomet,et al.  Key Range Locking Strategies for Improved Concurrency , 1993, VLDB.

[31]  Philip A. Bernstein,et al.  Concurrency Control in Distributed Database Systems , 1986, CSUR.

[32]  Michael Stonebraker,et al.  The End of an Architectural Era (It's Time for a Complete Rewrite) , 2007, VLDB.

[33]  Daniel J. Abadi,et al.  Calvin: fast distributed transactions for partitioned database systems , 2012, SIGMOD Conference.

[34]  Seif Haridi,et al.  Apache Flink™: Stream and Batch Processing in a Single Engine , 2015, IEEE Data Eng. Bull..

[35]  Daniel J. Abadi,et al.  Low overhead concurrency control for partitioned main memory databases , 2010, SIGMOD Conference.

[36]  Jignesh M. Patel,et al.  High-Performance Concurrency Control Mechanisms for Main-Memory Databases , 2011, Proc. VLDB Endow..

[37]  Divyakant Agrawal,et al.  Modular Synchronization in Distributed, Multiversion Databases: Version Control and Concurrency Control , 1993, IEEE Trans. Knowl. Data Eng..

[38]  Takashi Horikawa.  Latch-free data structures for DBMS: design, implementation, and evaluation , 2013, SIGMOD '13.

[39]  Ippokratis Pandis,et al.  Improving OLTP Scalability using Speculative Lock Inheritance , 2009, Proc. VLDB Endow..

[40]  Daniel J. Abadi,et al.  Modularity and Scalability in Calvin , 2013, IEEE Data Eng. Bull..

[41]  Arthur T. Whitney,et al.  High volume transaction processing without currency control, two phase commit, SQLor C++ , 1997 .