Taking Omid to the Clouds: Fast, Scalable Transactions for Real-Time Cloud Analytics

We describe how we evolve Omid, a transaction processing system for Apache HBase, to power Apache Phoenix, a cloud-grade real-time SQL analytics engine. Omid was originally designed for data processing pipelines at Yahoo, which are, by and large, throughput-oriented monolithic NoSQL applications. Providing a platform to support converged real-time transaction processing and analytics applications – dubbed translytics – introduces new functional and performance requirements. For example, SQL support is key for developer productivity, multi-tenancy is essential for cloud deployment, and latency is cardinal for just-in-time data ingestion and analytics insights. We discuss our efforts to adapt Omid to these new domains, as part of the process of integrating it into Phoenix as the transaction processing backend. A central piece of our work is latency reduction in Omid’s protocol, which also improves scalability. Under light load, the new protocol’s latency is 4x to 5x smaller than the legacy Omid’s, whereas under increased loads it is an order of magnitude faster. We further describe a fast path protocol for single-key transactions, which enables processing them almost as fast as native HBase operations. PVLDB Reference Format: Shacham, Gottesman, Bergman, Bortnikov, Hillel, and Keidar. Taking Omid to the Clouds: Fast, Scalable Transactions for Real-Time Cloud Analytics. PVLDB, 11 (12): 1795 1808, 2018. DOI: https://doi.org/10.14778/3229863.3229868

[1]  Eshcar Hillel,et al.  Reconciling Transactional and Non-Transactional Operations in Distributed Key-Value Stores , 2014, SYSTOR 2014.

[2]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[3]  Idit Keidar,et al.  Ordering Transactions with Prediction in Distributed Object Stores , 2013 .

[4]  Ian Rae,et al.  F1: A Distributed SQL Database That Scales , 2013, Proc. VLDB Endow..

[5]  Komal Shringare,et al.  Apache Hadoop Goes Realtime at Facebook , 2015 .

[6]  Marcos K. Aguilera,et al.  Yesquel: scalable sql storage for web applications , 2014, SOSP.

[7]  Emin Gün Sirer,et al.  HyperDex: a distributed, searchable key-value store , 2012, SIGCOMM '12.

[8]  Barbara Liskov,et al.  Granola: Low-Overhead Distributed Transaction Coordination , 2012, USENIX Annual Technical Conference.

[9]  Yawei Li,et al.  Megastore: Providing Scalable, Highly Available Storage for Interactive Services , 2011, CIDR.

[10]  Divyakant Agrawal,et al.  Serializability, not Serial: Concurrency Control and Availability in Multi-Datacenter Datastores , 2012, Proc. VLDB Endow..

[11]  Benjamin Reed,et al.  Omid: Lock-free transactional support for distributed data stores , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[12]  Haibo Chen,et al.  Fast In-Memory Transaction Processing Using RDMA and HTM , 2017, ACM Trans. Comput. Syst..

[13]  Miguel Castro,et al.  FaRM: Fast Remote Memory , 2014, NSDI.

[14]  Idit Keidar,et al.  Omid, Reloaded: Scalable and Highly-Available Transaction Processing , 2017, FAST.

[15]  Emin Gün Sirer,et al.  Warp: Lightweight Multi-Key Transactions for Key-Value Stores , 2015, ArXiv.

[16]  Miguel Castro,et al.  No compromises: distributed transactions with consistency, availability, and performance , 2015, SOSP.

[17]  Frank Dabek,et al.  Large-scale Incremental Processing Using Distributed Transactions and Notifications , 2010, OSDI.

[18]  Daniel J. Abadi,et al.  Calvin: fast distributed transactions for partitioned database systems , 2012, SIGMOD Conference.

[19]  Murat Demirbas,et al.  Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases , 2014 .

[20]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[21]  Kenneth Baclawski,et al.  Quickly generating billion-record synthetic databases , 1994, SIGMOD '94.

[22]  Tao Zou,et al.  Tango: distributed data structures over a shared log , 2013, SOSP.

[23]  Jim Gray,et al.  A critique of ANSI SQL isolation levels , 1995, SIGMOD '95.

[24]  Christopher Frost,et al.  Spanner: Google's Globally-Distributed Database , 2012, OSDI.

[25]  Fernando Pedone,et al.  Database replication using generalized snapshot isolation , 2005, 24th IEEE Symposium on Reliable Distributed Systems (SRDS'05).

[26]  Andreas Reuter,et al.  Transaction Processing: Concepts and Techniques , 1992 .