Dockerization Impacts in Database Performance Benchmarking

Docker seems to be an attractive solution for cloud database benchmarking as it simplifies the setup process through pre-built images that are portable and simple to maintain. However, the usage of Docker for benchmarking is only valid if there is no effect on measurement results. Existing work has so far only focused on the performance overheads that Docker directly induces for specific applications. In this paper, we have studied indirect effects of dockerization on the results of database benchmarking. Among others, our results clearly show that containerization has a measurable and non-constant influence on measurement results and should, hence, only be used after careful analysis.

[1]  Ramakrishnan Rajamony,et al.  An updated performance comparison of virtual machines and Linux containers , 2015, 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[2]  Sherif Sakr,et al.  Towards Comprehensive Measurement of Consistency Guarantees for Cloud-Hosted Data Storage Services , 2013, TPCTC.

[3]  Lewis Tseng,et al.  Characterizing and Adapting the Consistency-Latency Tradeoff in Distributed Key-Value Stores , 2015, ACM Trans. Auton. Adapt. Syst..

[4]  Prashant Malik,et al.  Cassandra: a decentralized structured storage system , 2010, OPSR.

[5]  Pablo Prieto,et al.  The impact of Docker containers on the performance of genomic pipelines , 2015, PeerJ.

[6]  Jóakim von Kistowski,et al.  How to Build a Benchmark , 2015, ICPE.

[7]  Nam Thoai,et al.  Using Docker in high performance computing applications , 2016, 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE).

[8]  David Bermbach,et al.  Cloud Service Benchmarking - Measuring Quality of Cloud Services from a Client Perspective , 2017 .

[9]  Hans-Arno Jacobsen,et al.  PNUTS: Yahoo!'s hosted data serving platform , 2008, Proc. VLDB Endow..

[10]  David Bermbach,et al.  A Runtime Quality Measurement Framework for Cloud Database Service Systems , 2012, 2012 Eighth International Conference on the Quality of Information and Communications Technology.

[11]  Alexandru Iosup,et al.  Benchmarking in the Cloud: What It Should, Can, and Cannot Be , 2012, TPCTC.

[12]  Blesson Varghese,et al.  Plug and play bench: Simplifying big data benchmarking using containers , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[13]  David Bermbach,et al.  Benchmarking the Performance Impact of Transport Layer Security in Cloud Database Systems , 2014, 2014 IEEE International Conference on Cloud Engineering.

[14]  Jorge Bernardino,et al.  Testing Cloud Benchmark Scalability with Cassandra , 2014, 2014 IEEE World Congress on Services.

[15]  Sherif Sakr,et al.  Towards an Extensible Middleware for Database Benchmarking , 2014, TPCTC.

[16]  Tilmann Rabl,et al.  Solving Big Data Challenges for Enterprise Application Performance Management , 2012, Proc. VLDB Endow..

[17]  Jörn Kuhlenkamp,et al.  Benchmarking Scalability and Elasticity of Distributed Database Systems , 2014, Proc. VLDB Endow..

[18]  Blesson Varghese,et al.  Container-Based Cloud Virtual Machine Benchmarking , 2016, 2016 IEEE International Conference on Cloud Engineering (IC2E).

[19]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[20]  David Bermbach,et al.  Is it safe to dockerize my database benchmark? , 2019, SAC.

[21]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.

[22]  Michael Ferdman,et al.  Demystifying cloud benchmarking , 2016, 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[23]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[24]  Karl Huppler,et al.  The Art of Building a Good Benchmark , 2009, TPCTC.

[25]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[26]  Adam Wierman,et al.  Open Versus Closed: A Cautionary Tale , 2006, NSDI.