Omid, Reloaded: Scalable and Highly-Available Transaction Processing

We present Omid - a transaction processing service that powers web-scale production systems at Yahoo. Omid provides ACID transaction semantics on top of traditional key-value storage; its implementation over Apache HBase is open sourced as part of Apache Incubator. Omid can serve hundreds of thousands of transactions per second on standard mid-range hardware, while incurring minimal impact on the speed of data access in the underlying key-value store. Additionally, as expected from always-on production services, Omid is highly available.

[1]  Marcos K. Aguilera,et al.  Yesquel: scalable sql storage for web applications , 2014, SOSP.

[2]  Emin Gün Sirer,et al.  Warp: Lightweight Multi-Key Transactions for Key-Value Stores , 2015, ArXiv.

[3]  Jim Gray,et al.  A critique of ANSI SQL isolation levels , 1995, SIGMOD '95.

[4]  Philip A. Bernstein,et al.  Hyder - A Transactional Record Manager for Shared Flash , 2011, CIDR.

[5]  David E. Culler,et al.  An architecture for highly concurrent, well-conditioned internet services , 2002 .

[6]  Tim Kraska,et al.  MDCC: multi-data center consistency , 2012, EuroSys '13.

[7]  Miguel Castro,et al.  FaRM: Fast Remote Memory , 2014, NSDI.

[8]  Andreas Reuter,et al.  Transaction Processing: Concepts and Techniques , 1992 .

[9]  Idit Keidar,et al.  Ordering Transactions with Prediction in Distributed Object Stores , 2013 .

[10]  Divyakant Agrawal,et al.  Serializability, not Serial: Concurrency Control and Availability in Multi-Datacenter Datastores , 2012, Proc. VLDB Endow..

[11]  Fernando Pedone,et al.  Sprint: a middleware for high-performance transaction processing , 2007, EuroSys '07.

[12]  Arvind Krishnamurthy,et al.  Building consistent transactions with inconsistent replication , 2015, SOSP.

[13]  Tao Zou,et al.  Tango: distributed data structures over a shared log , 2013, SOSP.

[14]  James A. Cowling.  Low-overhead distributed transaction coordination , 2012 .

[15]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[16]  Christopher Frost,et al.  Spanner: Google's Globally-Distributed Database , 2012, OSDI.

[17]  Daniel Gómez Ferro,et al.  A critique of snapshot isolation , 2012, EuroSys '12.

[18]  Leslie Lamport,et al.  The part-time parliament , 1998, TOCS.

[19]  Frank Dabek,et al.  Large-scale Incremental Processing Using Distributed Transactions and Notifications , 2010, OSDI.

[20]  Michael J. Cahill.  Serializable isolation for snapshot databases , 2009, TODS.

[21]  Daniel J. Abadi,et al.  Calvin: fast distributed transactions for partitioned database systems , 2012, SIGMOD Conference.

[22]  Haibo Chen,et al.  Fast In-Memory Transaction Processing Using RDMA and HTM , 2017, ACM Trans. Comput. Syst..

[23]  Miguel Castro,et al.  No compromises: distributed transactions with consistency, availability, and performance , 2015, SOSP.

[24]  Dennis Shasha,et al.  The dangers of replication and a solution , 1996, SIGMOD '96.

[25]  Marcos K. Aguilera,et al.  Sinfonia: a new paradigm for building scalable distributed systems , 2007, SOSP.

[26]  Dennis Shasha,et al.  Making snapshot isolation serializable , 2005, TODS.

[27]  Barbara Liskov,et al.  Granola: Low-Overhead Distributed Transaction Coordination , 2012, USENIX Annual Technical Conference.

[28]  Yawei Li,et al.  Megastore: Providing Scalable, Highly Available Storage for Interactive Services , 2011, CIDR.

[29]  Dahlia Malkhi,et al.  CORFU: A distributed shared log , 2013, TOCS.

[30]  Emin Gün Sirer,et al.  HyperDex: a distributed, searchable key-value store , 2012, SIGCOMM '12.

[31]  Benjamin Reed,et al.  Omid: Lock-free transactional support for distributed data stores , 2014, 2014 IEEE 30th International Conference on Data Engineering.