Research on a Clustering Data De-Duplication Mechanism Based on Bloom Filter

Recently, data de-duplication, the hot emerging technology, has received a broad attention from both academia and industry. Some researches focus on the approach by which to reduce more redundant data. And the others investigate how to do de-duplication at high speed. In this paper, we aim at reducing the time and space requirement for data de-duplication. We describe a clustering architecture with multiple nodes and all nodes can do the chunk-level data de-duplication in parallel. Thus the performance will be improved noticeably. At the same time, this paper proposes a new technique called "Fingerprint Summary". Each node keeps a compact summary of the chunks' fingerprints of every other node in its memory. When checking for duplicate chunks, each node queries its local chunk hash database and then the Fingerprint Summary if necessary to eliminate inter-node redundant chunks. So we can reduce the storage capacity requirement largely.