Exploring the Use of Elastic Resource Federations for Enabling Large-Scale Scientific Workflows

An important class of scientific and engineering workflows, e.g. those used for uncertainty quantification, design optimization and parametric studies, naturally map onto the Many-Task Computing (MTC) paradigm. However, what distinguishes these workloads is a unique combination of dynamically changing resource requirements and very large computational and throughput demands. Such workflows can benefit from an elastic execution infrastructure that is based on the dynamic federation of resources. The overarching goal of this paper is to explore the nature of such an elastic, dynamically federated platform, and to experimentally demonstrate that it can effectively support the targeted class of scientific and engineering workflows. As a driving application for our study we use the problem of constructing a phase diagram in microfluidics, which is representative for a broader class of parameter space interrogation techniques. To satisfy its computational demands of 2.5 million corehours within reasonable time limits, we construct a dynamic federation of ten HPC resources from six different computing centers. This experiment delivers the most comprehensive data on fluid flow in a microchannel with an obstacle. Moreover, it offers important insights that enable us to identify key requirements and architectural components that a platform based on federated resources must provide in order to efficiently handle considered scientific MTC workloads.