From Galactic archeology to soil metagenomics - surfing on massive data streams.

Soil microbiologists make their discovery in the dirt and rarely look at the stars. The words of Leonardo da Vinci ‘We know more about the movement of celestial bodies than about the soil underfoot’ are as applicable now as they were in 1510 and, currently, scientists deciphering soil microbe genomes and exploring the metagenomes of soil ecosystems may learn from their sky-gazing colleagues. Keeping their feet in the mud, but having their head in the sky, may help them to avoid the (meta)genome-analysis gridlock. Metagenomics involves sampling and sequencing the genome sequences of a community of organisms that inhabit a common environment, such as the ocean, the soil or the human gut (Handelsman, 2004; Hugenholtz & Tyson, 2008). Metagenomics provides an unbiased picture of the community structure (species richness and distribution) and its functional potential. It is rapidly moving from being a description tool to an experimental tool as a result of comparisons now being made of metagenomes submitted to environmental perturbations. Soils are home to microbial communities whose aggregate membership (5 · 10) (Whitman et al., 1998) outnumbers the stars shining in the sky (7 · 10). Soil complex habitats contain an estimated 10 to 10 species in a single gram (Gans et al., 2005; Curtis & Sloan, 2006). Once the genomes of these hundreds of thousands of species that crawl on weathered rocks, decaying organic matter and in the rhizosphere are catalogued, the streams of data arising from soil will equal those pouring from star-gazing telescopes. However, a major problem has taken the microbiologist community offguard: how to analyze this exponentially increasing amount of sequence data. As quoted from Kahvejian et al. (2008) ‘This surge of new data can be received as a flood, overwhelming the unsuspecting researcher, or as a tremendous wave that can be surfed to new horizons.’ Soil (meta)genomics sprang from fast advances in sequencing technology, and continued improvements are providing data in quantities unimaginable a few years ago. In the coming years, mining a day’s worth of data will take more than a day of large supercomputer time, and the fraction of the available data that we will be able to mine and analyze in a useful period of time will rapidly dwindle towards zero. Overcoming this bottleneck requires a shift in our frame of mind from mining databases to mining data streams. Recent papers (Singh et al., 2009) have highlighted the needs, requirements and challenges for sequencing microbial genomes and soil metagenomes in line with the Human Microbiome Project (http://nihroadmap.nih.gov/hmp/) and the Global Ocean Sampling Expedition (Rusch et al., 2007). Before discussing how other scientific communities are navigating through the hazards of massive data streams, we will summarize the current status of genomics and metagenomics of soil micro-organisms.