An Efficient Design and Implementation of Deduplication on Open-Channel SSDs

Since the number of NAND flash write cycles reaches the upper limit, data cannot be reliably stored. Therefore, the lifespan of Solid State Disks (SSDs) is limited, making it impossible to become a reliable storage device in a write-intensive data center. Deduplication can effectively reduce write traffic and increase the lifespan of SSDs. However, the fingerprint calculation overhead in the traditional data deduplication scheme is too large, which will reduce the write performance of the SSDs. In this paper, we propose a new deduplication scheme D-pblk based on Open-Channel SSDs(OCSSDs), which can reduce fingerprint computing overhead with modern high performance CPUs. In D-pblk, we use the light-weight hash CRC32 to filter most of the non-duplicated data, and When CRC32 is first hit, we read the data from Open-Channel SSDs and calculate the heavy-weight hash SHA-1 for accurate comparison, which reducing fingerprint computing overhead and hardly affects the deduplication rate of workloads. And we use multi-thread to calculate fingerprints in parallel, further reducing fingerprint computing overhead. In addition, we use double ring buffers to solve the problem that the data size after deduplication may not match the size of a flash page. We use FEMU to simulate Open-Channel SSDs for evaluation. The experimental results show that D-pblk can achieve the deduplication rate ranging from 4.61% to 31.63% and the write latency is improved by a factor of up to 29%.