Sub-linear Sequence Search via a Repeated And Merged Bloom Filter (RAMBO)

Whole-genome shotgun sequencing (WGS), especially that of microbial genomes, has been the core of recent research advances in large-scale comparative genomics. The data deluge has resulted in exponential growth in genomic datasets over the past years and has shown no sign of slowing down. Several recent attempts have been made to tame the computational burden of read classification and sequence search on these ultra large-scale datasets, including both raw reads and assembled genomes. A notable recent method is BigSI. BigSI is based around bloom filters and offers very efficient query sequence search times. However, querying with BigSI still requires probing Bloom filters (or sets of bitslices) which scales linearly with the number of datasets. As a result, scaling up BigSI for datasets with potentially millions (or higher) samples is likely prohibitive. In this paper, we propose RAMBO (Repeated and Merged Bloom Filter) where the number of Bloom filter probes is significantly less than BigSI due to sub-linear scaling for the same false-positive rate. Our idea is theoretically sound and inspired by the count-min sketch data structure, a popular streaming algorithm. RAMBO provides a significant improvement over BigSI in terms of query time when evaluated on real genome datasets. Furthermore, due to sub-linear scaling, the larger the size and number of datasets, the bigger the gains are with RAMBO over BigSI.