DREAM-Yara: An implementation of an exact read mapper for very large databases

Motivation: Mapping-based approaches have become limited in their application to very large sets of references since computing an FM-index for very large databases (e.g. > 10 GB) has become a bottleneck. This affects many analyses that need this index as an essential step for approximate matching of the NGS reads to reference databases. For instance, in metagenomics, the size of the reference sequences has become prohibitive to compute a single full-text index on standard machines. Even on large memory machines, computing the index takes about one day of compute time. As a result, updates of indices are rarely performed. Hence, it is desirable to distribute the indices to solve the index construction and update problem while preserving fast search times. Results: To solve the index construction and update problem we propose the DREAM (Dynamic seaRchablE pArallel coMpressed index) framework in this paper and provide an implementation. The main contributions are the introduction binning directories using a novel use of Bloom filters. We combine several Bloom filters to form an interleaved Bloom filter and use this new data structure to quickly exclude reads for parts of the databases where they cannot match. This allows us to keep the databases in several indices which can be easily rebuild if parts are updated while maintaining a fast search time. The second main contribution lies in the integration of the Yara exact read mapper (Siragusa, 2013) in a distributed version for the DREAM framework.