Performance Optimization of Large Files Writes to Ceph Based on Multiple Pipelines Algorithm

As a cloud storage platform software, Ceph can be used to obtain petabyte-scale storage system built from commodity hardware [1] [2] [3]. In the previous work, we optimized the performance of reads/writes from/to the Ceph storage cluster based on multi-threaded algorithms [4]. Experiment results indicate that the performance of small files reads/writes and large files reads algorithms improve obviously, but the performance of the large files writes to Ceph by the single pipeline algorithm has no evident improvement. To address this problem, we use multiple pipelines algorithm to optimize the performance of the large files writes to the Ceph storage cluster. The experiment results show that when the size of the data block is set to 10MB, the maximal performance improvement percentage of the multiple pipelines algorithm running on the two logical CPUs machine is 100.70%. At the same time, limited by the multi-threaded mechanism of Python language, such as Python GIL and threads thrashing, the performance of the multiple pipelines algorithm running on multiple cores machines does not increase linearly. We intend to optimize the performance of large files writes algorithm using C++ version application program interface of Ceph for the future work.