Data Management: State-of-the-Practice at Open-Science Data Centers

Recent advancements in the field of instrumentation and the computational breakthroughs have enabled researchers to gather increasingly large datasets. In order to efficiently process, analyze, and store such large datasets, High Performance Computing (HPC) systems are needed. Various institutions provide HPC resources and services to the scientific community for doing open-science research, often without any direct cost to the community. However, maintaining large datasets (in the range of TBs and above) for in-place processing and analyses is usually not an option in open-science data centers because the resources are shared amongst multiple projects and the disk-space owned by any user-account is limited. Therefore, for storage of large volumes of data that could be of short-term, mid-term or long-term retention value, large capacity online disks and/or tape-based archival systems are used. The data is moved between storage and computational resources as needed. Given such a scenario, optimal strategies for data movement, data archiving, and data preservation are crucial. In this chapter, we discuss state-of-the-practice in data storage infrastructure, data movement, data archiving, and data preservation at open-science data centers supporting data-intensive computing.