A Fast Dual-level Fingerprinting Scheme for Data Deduplication

Data deduplication has attracted recent interest in the research community. Several approaches are proposed to eliminate duplicate data first at the file level and then at the chunk level to reduce the duplicate-lookup complexity. To meet the high-throughput requirements, this paper proposes a fast dual-level fingerprinting (FDF) scheme that can fingerprint a dataset both at the file level and at the chunk level in a single scan of the contents. FDF breaks the fingerprinting process into task segments and further leverage the computing resources of modern multi-core CPUs to pipeline the time-consuming operations. The proposed FDF scheme has been evaluated in an experimental data backup network with real-world datasets and compared with an alternative two-stage approach. Experimental results show that FDF can maintain over 100MB/s fingerprinting throughput that matches the bandwidth of a gigabit network adapter while being fully pipelined.