Speedy transactions in multicore in-memory databases

Silo is a new in-memory database that achieves excellent performance and scalability on modern multicore machines. Silo was designed from the ground up to use system memory and caches efficiently. For instance, it avoids all centralized contention points, including that of centralized transaction ID assignment. Silo's key contribution is a commit protocol based on optimistic concurrency control that provides serializability while avoiding all shared-memory writes for records that were only read. Though this might seem to complicate the enforcement of a serial order, correct logging and recovery is provided by linking periodically-updated epochs with the commit protocol. Silo provides the same guarantees as any serializable database without unnecessary scalability bottlenecks or much additional latency. Silo achieves almost 700,000 transactions per second on a standard TPC-C workload mix on a 32-core machine, as well as near-linear scalability. Considered per core, this is several times higher than previously reported results.

[1]  Irving L. Traiger,et al.  The notions of consistency and predicate locks in a database system , 1976, CACM.

[2]  J. T. Robinson,et al.  On optimistic methods for concurrency control , 1979, TODS.

[3]  S. B. Yao,et al.  Efficient locking for concurrent operations on B-trees , 1981, TODS.

[4]  C. Mohan,et al.  ARIES/KVL: A Key-Value Locking Method for Concurrency Control of Multiaction Transactions Operating on B-Tree Indexes , 1990, VLDB.

[5]  Rainer Unland,et al.  Optimistic concurrency control revisited , 1994 .

[6]  Jim Gray,et al.  A critique of ANSI SQL isolation levels , 1995, SIGMOD '95.

[7]  Kihong Kim,et al.  Cache-Conscious Concurrency Control of Main-Memory Indexes on Shared-Memory Multiprocessor Systems , 2001, VLDB.

[8]  Thki Hder,et al.  OBSERVATIONS ON OPTIMISTIC CONCURRENCY CONTROL SCHEMES , 2003 .

[9]  Keir Fraser,et al.  Practical lock-freedom , 2003 .

[10]  Divyakant Agrawal,et al.  Distributed optimistic concurrency control with reduced rollback , 2005, Distributed Computing.

[11]  Nir Shavit,et al.  Transactional Locking II , 2006, DISC.

[12]  Michael Stonebraker,et al.  The End of an Architectural Era (It's Time for a Complete Rewrite) , 2007, VLDB.

[13]  Jonathan Walpole,et al.  Performance of memory reclamation for lockless synchronization , 2007, J. Parallel Distributed Comput..

[14]  Babak Falsafi,et al.  Shore-MT: a scalable storage manager for the multicore era , 2009, EDBT '09.

[15]  Ippokratis Pandis,et al.  Data-oriented transaction execution , 2010, Proc. VLDB Endow..

[16]  Daniel J. Abadi,et al.  Low overhead concurrency control for partitioned main memory databases , 2010, SIGMOD Conference.

[17]  Kunle Olukotun,et al.  A practical concurrent binary search tree , 2010, PPoPP '10.

[18]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[19]  Ippokratis Pandis,et al.  PLP: Page Latch-free Shared-everything OLTP , 2011, Proc. VLDB Endow..

[20]  Jignesh M. Patel,et al.  High-Performance Concurrency Control Mechanisms for Main-Memory Databases , 2011, Proc. VLDB Endow..

[21]  Pradeep Dubey,et al.  PALM: Parallel Architecture-Friendly Latch-Free Modifications to B+ Trees on Many-Core Processors , 2011, Proc. VLDB Endow..

[22]  Gustavo Alonso,et al.  Database engines on multicores, why parallelize when you can distribute? , 2011, EuroSys '11.

[23]  Daniel J. Abadi,et al.  Lightweight Locking for Main Memory Database Systems , 2012, Proc. VLDB Endow..

[24]  Christopher Frost,et al.  Spanner: Google's Globally-Distributed Database , 2012, OSDI.

[25]  Carlo Curino,et al.  Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems , 2012, SIGMOD Conference.

[26]  Ippokratis Pandis,et al.  OLTP on Hardware Islands , 2012, Proc. VLDB Endow..

[27]  Eddie Kohler,et al.  Cache craftiness for fast multicore key-value storage , 2012, EuroSys '12.

[28]  Sudipta Sengupta,et al.  The Bw-Tree: A B-tree for new hardware platforms , 2013, 2013 IEEE 29th International Conference on Data Engineering (ICDE).

[29]  Craig Freedman,et al.  Hekaton: SQL server's memory-optimized OLTP engine , 2013, SIGMOD '13.

[30]  Sudipta Sengupta,et al.  LLAMA: A Cache/Storage Subsystem for Modern Hardware , 2013, Proc. VLDB Endow..

[31]  Heon Young Yeom,et al.  A scalable lock manager for multicores , 2013, SIGMOD '13.

[32]  M. Frans Kaashoek,et al.  RadixVM: scalable address spaces for multithreaded applications , 2013, EuroSys '13.

[33]  Takashi Horikawa.  Latch-free data structures for DBMS: design, implementation, and evaluation , 2013, SIGMOD '13.