Quotient hash tables: efficiently detecting duplicates in streaming data

This article presents the Quotient Hash Table (QHT) a new data structure for duplicate detection in unbounded streams. QHTs stem from a corrected analysis of streaming quotient filters (SQFs), resulting in a 33% reduction in memory usage for equal performance. We provide a new and thorough analysis of both algorithms, with results of interest to other existing constructions. We also introduce an optimised version of our new data structure dubbed Queued QHT with Duplicates (QQHTD). We prove in a benchmark that QHT and QQHTD both are, at the same time more efficient and faster than any other filter from the literature, by a large margin. Finally we discuss the effect of adversarial inputs for hash-based duplicate filters similar to QHT.