Behaviors of Storage Backends in Ceph Object Store

Ceph is a scalable, reliable and high-performance storage solution that is widely used in the cloud computing environment. Internally, Ceph provides three different storage backends: FileStore, KStore and BlueStore. However, little effort has been devoted to identifying the differences in those storage backends and their implications on performance. In this paper, we carry out extensive analysis with a microbenchmark and a long-term workload to compare Ceph storage backends and understand their write behaviors by focusing on WAF (Write Amplification Factor). To accurately analyze WAF, we carefully classify write traffic into several categories for each storage backend. We find that writes are amplified by more than 13x, no matter which Ceph storage backend is used. In FileStore, the overhead of Ceph write-ahead journaling triples write traffic compared to the original data size. Also, FileStore has the journaling of journal problem, generating a relatively large amount of file system metadata and journal traffic. KStore suffers severe fluctuations in IOPS (I/O Operations Per Second) and WAF due to large compaction overheads. BlueStore shows the stable performance on both HDDs and SSDs in terms of IOPS, WAF and latency. Overall, FileStore performs the best among all storage backends on SSDs, while BlueStore is also highly promising with good average and tail latency even on HDDs.