Using restricted transactional memory to build a scalable in-memory database

The recent availability of Intel Haswell processors marks the transition of hardware transactional memory from research toys to mainstream reality. DBX is an in-memory database that uses Intel's restricted transactional memory (RTM) to achieve high performance and good scalability across multi-core machines. The main limitation (and also key to practicality) of RTM is its constrained working set size: an RTM region that reads or writes too much data will always be aborted. The design of DBX addresses this challenge in several ways. First, DBX builds a database transaction layer on top of an underlying shared-memory store. The two layers use separate RTM regions to synchronize shared memory access. Second, DBX uses optimistic concurrency control to separate transaction execution from its commit. Only the commit stage uses RTM for synchronization. As a result, the working set of the RTMs used scales with the meta-data of reads and writes in a database transaction as opposed to the amount of data read/written. Our evaluation using TPC-C workload mix shows that DBX achieves 506,817 transactions per second on a 4-core machine.

[1]  Eddie Kohler,et al.  Cache craftiness for fast multicore key-value storage , 2012, EuroSys '12.

[2]  Hector Garcia-Molina,et al.  Main Memory Database Systems: An Overview , 1992, IEEE Trans. Knowl. Data Eng..

[3]  Alfons Kemper,et al.  HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[4]  Jeffrey F. Naughton,et al.  On Transactional Memory, Spinlocks, and Database Transactions , 2010, ADMS@VLDB.

[5]  Gustavo Alonso,et al.  Database engines on multicores, why parallelize when you can distribute? , 2011, EuroSys '11.

[6]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[7]  Babak Falsafi,et al.  Shore-MT: a scalable storage manager for the multicore era , 2009, EDBT '09.

[8]  Mark Moir,et al.  Simplifying concurrent algorithms by exploiting hardware transactional memory , 2010, SPAA '10.

[9]  Don S. Batory,et al.  GENESIS: An Extensible Database Management System , 1988, IEEE Trans. Software Eng..

[10]  Eric A. Brewer,et al.  Stasis: flexible transactional storage , 2006, OSDI '06.

[11]  C. Mohan,et al.  ARIES/KVL: A Key-Value Locking Method for Concurrency Control of Multiaction Transactions Operating on B-Tree Indexes , 1990, VLDB.

[12]  Robert E. Tarjan,et al.  Deletion without rebalancing in balanced binary trees , 2010, SODA '10.

[13]  Haibo Chen,et al.  Opportunities and pitfalls of multi-core scaling using hardware transaction memory , 2013, APSys.

[14]  Eric Ruppert,et al.  Lock-free linked lists and skip lists , 2004, PODC '04.

[15]  Irving L. Traiger,et al.  The notions of consistency and predicate locks in a database system , 1976, CACM.

[16]  Jignesh M. Patel,et al.  High-Performance Concurrency Control Mechanisms for Main-Memory Databases , 2011, Proc. VLDB Endow..

[17]  Times-Ten Team,et al.  In-memory data management for consumer transactions the timesten approach , 1999, SIGMOD '99.

[18]  Craig Freedman,et al.  Hekaton: SQL server's memory-optimized OLTP engine , 2013, SIGMOD '13.

[19]  Mark Moir,et al.  Early experience with a commercial hardware transactional memory implementation , 2009, ASPLOS.

[20]  Eddie Kohler,et al.  Modular data storage with Anvil , 2009, SOSP '09.

[21]  Viktor Leis,et al.  Exploiting hardware transactional memory in main-memory databases , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[22]  Nir Shavit,et al.  Transactional Locking II , 2006, DISC.

[23]  Michael Stonebraker,et al.  H-store: a high-performance, distributed main memory transaction processing system , 2008, Proc. VLDB Endow..

[24]  S. B. Yao,et al.  Efficient locking for concurrent operations on B-trees , 1981, TODS.

[25]  J. T. Robinson,et al.  On optimistic methods for concurrency control , 1979, TODS.

[26]  Milo M. K. Martin,et al.  Subtleties of transactional memory atomicity semantics , 2006, IEEE Computer Architecture Letters.

[27]  Carlo Curino,et al.  Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems , 2012, SIGMOD Conference.

[28]  James R. Goodman,et al.  Speculative lock elision: enabling highly concurrent multithreaded execution , 2001, MICRO.

[29]  Marc Tremblay,et al.  Rock: A High-Performance Sparc CMT Processor , 2009, IEEE Micro.

[30]  Maurice Herlihy,et al.  Transactional Memory: Architectural Support For Lock-free Data Structures , 1993, Proceedings of the 20th Annual International Symposium on Computer Architecture.

[31]  Eddie Kohler,et al.  Speedy transactions in multicore in-memory databases , 2013, SOSP.

[32]  Hamid Pirahesh,et al.  A data management extension architecture , 1987, SIGMOD '87.

[33]  Mark Moir,et al.  Using hardware transactional memory to correct and simplify and readers-writer lock algorithm , 2013, PPoPP '13.

[34]  Paul E. McKenney,et al.  Making lockless synchronization fast: performance implications of memory reclamation , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[35]  Peter Boncz,et al.  UvA-DARE ( Digital Academic Repository ) Monet ; a next-Generation DBMS Kernel For Query-Intensive Applications , 2007 .

[36]  Daniel J. Abadi,et al.  Modularity and Scalability in Calvin , 2013, IEEE Data Eng. Bull..

[37]  Sean White,et al.  Hybrid NOrec: a case study in the effectiveness of best effort hardware transactional memory , 2011, ASPLOS XVI.

[38]  Haibo Chen,et al.  SSMalloc: a low-latency, locality-conscious memory allocator with stable performance scalability , 2012, APSys.

[39]  Nir Shavit,et al.  Reduced hardware transactions: a new approach to hybrid transactional memory , 2013, SPAA.

[40]  Michael Stonebraker,et al.  The End of an Architectural Era (It's Time for a Complete Rewrite) , 2007, VLDB.