Pangeo: A Big-data Ecosystem for Scalable Earth System Science

Data intensive scientific workflows are at a pivotal time in which traditional local computing resources are no longer capable of meeting the storage or computing demands of scientists. In the Earth System Sciences (ESS) community, we are facing an explosion of data volumes where new datasets, sourced from models, in-situ observations, and remote sensing platforms, are being made available at prohibitively large volumes to store and work with using conventional computational approaches. Furthermore, there is a growing recognition that the fragmentation of software tools and environments renders most geoscientific research effectively unreproducible and prone to failure. While the private data-science community has been actively developing big-data and opensource solutions to these problems, the academic community has begun to lag, leading to a growing technology gap between the two sectors.