Data structures and compression algorithms for genomic sequence data

MOTIVATION The continuing exponential accumulation of full genome data, including full diploid human genomes, creates new challenges not only for understanding genomic structure, function and evolution, but also for the storage, navigation and privacy of genomic data. Here, we develop data structures and algorithms for the efficient storage of genomic and other sequence data that may also facilitate querying and protecting the data. RESULTS The general idea is to encode only the differences between a genome sequence and a reference sequence, using absolute or relative coordinates for the location of the differences. These locations and the corresponding differential variants can be encoded into binary strings using various entropy coding methods, from fixed codes such as Golomb and Elias codes, to variables codes, such as Huffman codes. We demonstrate the approach and various tradeoffs using highly variables human mitochondrial genome sequences as a testbed. With only a partial level of optimization, 3615 genome sequences occupying 56 MB in GenBank are compressed down to only 167 KB, achieving a 345-fold compression rate, using the revised Cambridge Reference Sequence as the reference sequence. Using the consensus sequence as the reference sequence, the data can be stored using only 133 KB, corresponding to a 433-fold level of compression, roughly a 23% improvement. Extensions to nuclear genomes and high-throughput sequencing data are discussed. AVAILABILITY Data are publicly available from GenBank, the HapMap web site, and the MITOMAP database. Supplementary materials with additional results, statistics, and software implementations are available from http://mammag.web.uci.edu/bin/view/Mitowiki/ProjectDNACompression.

[1]  Eran Halperin,et al.  Hinds , in Three Human Populations Whole-Genome Patterns of Common DNA Variation , 2008 .

[2]  R. Service.  The Race for the $1000 Genome , 2006, Science.

[3]  J. Lupski,et al.  The complete genome of an individual by massively parallel DNA sequencing , 2008, Nature.

[4]  Peter Elias,et al.  Universal codeword sets and representations of the integers , 1975, IEEE Trans. Inf. Theory.

[5]  C. Feschotte,et al.  DNA transposons and the evolution of eukaryotic genomes. , 2007, Annual review of genetics.

[6]  S. Harihara,et al.  Frequency of a 9-bp deletion in the mitochondrial DNA among Asian populations. , 1992, Human biology.

[7]  David B. Goldstein,et al.  Genomics: Understanding human diversity , 2005, Nature.

[8]  D. Turnbull,et al.  Reanalysis and revision of the Cambridge reference sequence for human mitochondrial DNA , 1999, Nature Genetics.

[9]  Pierre Baldi,et al.  MITOMASTER: a bioinformatics tool for the analysis of mitochondrial DNA sequences , 2009, Human mutation.

[10]  Bin Ma,et al.  DNACompress: fast and effective DNA sequence compression , 2002, Bioinform..

[11]  Henrik Bjørn Nielsen,et al.  OligoWiz 2.0—integrating sequence feature annotation into the design of microarray probes , 2005, Nucleic Acids Res..

[12]  Solomon W. Golomb,et al.  Run-length encodings (Corresp.) , 1966, IEEE Trans. Inf. Theory.

[13]  Pierre Baldi,et al.  Lossless Compression of Chemical Fingerprints Using Integer Entropy Codes Improves Storage and Retrieval , 2007, J. Chem. Inf. Model..

[14]  Shamkant B. Navathe,et al.  MITOMAP: a human mitochondrial genome database—2004 update , 2004, Nucleic Acids Res..

[15]  Marty C. Brandon,et al.  Natural selection shaped regional mtDNA variation in humans , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Zhaohui S. Qin,et al.  A second generation human haplotype map of over 3.1 million SNPs , 2007, Nature.

[17]  Timothy B. Stockwell,et al.  The Diploid Genome Sequence of an Individual Human , 2007, PLoS biology.

[18]  Glen G. Langdon,et al.  Arithmetic Coding , 1979 .

[19]  David A. Huffman,et al.  A method for the construction of minimum-redundancy codes , 1952, Proceedings of the IRE.

[20]  Geoffrey B. Nilsen,et al.  Whole-Genome Patterns of Common DNA Variation in Three Human Populations , 2005, Science.

[21]  D. Huffman.  A Method for the Construction of Minimum-Redundancy Codes , 1952 .

[22]  Alistair Moffat,et al.  Binary codes for locally homogeneous sequences , 2006, Inf. Process. Lett..

[23]  Toshihiro Tanaka.  The International HapMap Project , 2003, Nature.

[24]  S. Golomb.  Run-length encodings. , 1966 .

[25]  Behshad Behzadi,et al.  DNA Compression Challenge Revisited: A Dynamic Programming Approach , 2005, CPM.

[26]  Jocelyn Kaiser,et al.  A Plan to Capture Human Diversity in 1000 Genomes , 2008, Science.

[27]  Robert J. McEliece,et al.  The Theory of Information and Coding , 1979 .

[28]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[29]  E. Hagelberg,et al.  Molecular instability in the COII-tRNA(Lys) intergenic region of the human mitochondrial genome: multiple origins of the 9-bp deletion and heteroplasmy for expanded repeats. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[30]  Shamkant B. Navathe,et al.  MITOMAP: a human mitochondrial genome database--1998 update , 1998, Nucleic Acids Res..

[31]  Hugh E. Williams,et al.  Compression of nucleotide databases for fast searching , 1997, Comput. Appl. Biosci..

[32]  Xiaohui Xie,et al.  Sequence analysis Human genomes as email attachments , 2022 .

[33]  Jian Wang,et al.  The YH database: the first Asian diploid genome database , 2008, Nucleic Acids Res..

[34]  Ian H. Witten,et al.  Arithmetic coding for data compression , 1987, CACM.

[35]  E. Eichler,et al.  Fine-scale structural variation of the human genome , 2005, Nature Genetics.

[36]  Alistair Moffat,et al.  Binary Interpolative Coding for Effective Index Compression , 2000, Information Retrieval.

[37]  Dawei Li,et al.  The diploid genome sequence of an Asian individual , 2008, Nature.

[38]  A. Mortazavi,et al.  Genome-Wide Mapping of in Vivo Protein-DNA Interactions , 2007, Science.

[39]  Pierre Baldi,et al.  An enhanced MITOMAP with a global mtDNA mutational phylogeny , 2006, Nucleic Acids Res..

[40]  Pierre Baldi,et al.  Effective Compression of Monotone and Quasi-Monotone Sequences of Integers , 2008, Data Compression Conference (dcc 2008).

[41]  F. Sanger,et al.  Sequence and organization of the human mitochondrial genome , 1981, Nature.