NoSQL OLTP Benchmarking: A Survey

In recent years, various distributed NoSQL datastores have been developed that offer horizontal scalability and higher availability than traditional relational databases, but fewer querying options and only reduced consistency guarantees. The diversity of the design space makes it difficult to understand the performance implications of individual system designs. Existing benchmarking tools measure some relevant aspects, but do not capture all of them. In this paper, we give an overview of the state-of-the-art in NoSQL OLTP benchmarking, identify missing features as well as open challenges and point towards possible solutions.

[1]  Xiaozhou Li,et al.  Analyzing consistency properties for fun and profit , 2011, PODC '11.

[2]  Ion Stoica,et al.  Probabilistically Bounded Staleness for Practical Partial Quorums , 2012, Proc. VLDB Endow..

[3]  Alan Fekete,et al.  Quantifying Isolation Anomalies , 2009, Proc. VLDB Endow..

[4]  David Bermbach,et al.  Benchmarking Eventual Consistency: Lessons Learned from Long-Term Experimental Studies , 2014, 2014 IEEE International Conference on Cloud Engineering.

[5]  Sherif Sakr,et al.  Towards Comprehensive Measurement of Consistency Guarantees for Cloud-Hosted Data Storage Services , 2013, TPCTC.

[6]  Kevin Lee,et al.  Data Consistency Properties and the Trade-offs in Commercial Cloud Storage: the Consumers' Perspective , 2011, CIDR.

[7]  David Bermbach,et al.  Eventual consistency: How soon is eventual? An evaluation of Amazon S3's consistency behavior , 2011, MW4SOC '11.

[8]  Shahram Ghandeharizadeh,et al.  BG: A Benchmark to Evaluate Interactive Social Networking Actions , 2013, CIDR.

[9]  Wojciech M. Golab,et al.  Toward a Principled Framework for Benchmarking Consistency , 2012, HotDep.

[10]  Michael J. Freedman,et al.  Don't settle for eventual: scalable causal consistency for wide-area storage with COPS , 2011, SOSP.

[11]  Mahadev Konar,et al.  ZooKeeper: Wait-free Coordination for Internet-scale Systems , 2010, USENIX ATC.

[12]  Lin Xiao,et al.  YCSB++: benchmarking and performance debugging advanced features in scalable table stores , 2011, SoCC.

[13]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[14]  Ali Ghodsi,et al.  Highly Available Transactions: Virtues and Limitations , 2013, Proc. VLDB Endow..

[15]  Daniel J. Abadi,et al.  Consistency Tradeoffs in Modern Distributed Database System Design: CAP is Only Part of the Story , 2012, Computer.

[16]  Yves Le Traon,et al.  Under Pressure Benchmark for DDBMS Availability , 2013, J. Inf. Data Manag..

[17]  Ali Ghodsi,et al.  Bolt-on causal consistency , 2013, SIGMOD '13.

[18]  Alan Fekete,et al.  YCSB+T: Benchmarking web-scale transactional databases , 2014, 2014 IEEE 30th International Conference on Data Engineering Workshops.

[19]  Maurice Herlihy,et al.  Linearizability: a correctness condition for concurrent objects , 1990, TOPL.

[20]  Nancy A. Lynch,et al.  Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services , 2002, SIGA.