HPC-cloud native framework for concurrent simulation, analysis and visualization of CFD workflows

Abstract Analysis and rendering of high-resolution computational fluid dynamics (CFD) simulations often requires execution of multiple parallel data-processing pipelines. In this paper we present a hybrid cloud solution for efficient simulation and analysis of drop dispersions. The simulation component runs on a high-performance computing (HPC) cluster and cloud native framework performs the processing of CFD outputs. To facilitate some of the cloud features, we broke down the CFD data analysis pipeline into small microservice-like processes. In this way, a given resource can be set-up and shared between different simulations, codes and locations to minimize idle times and dynamically adjust the capacity to the workload while producing results on the fly, independent of the simulation engine. A combination of HPC and cloud technologies allows us to create agile and highly scalable solutions for CFD purposes. We focus on HPC-cloud infrastructure; however, other arrangements such as cloud-cloud or desktop-cloud can be easily adapted depending on the needs. This enables a framework for centralizing services for collaborative use between different users as well as for automatically providing data from different sources to machine learning algorithms. We describe a proof of concept implementation of the proposed framework and provide detailed analysis of its performance applied to a real two-phase flow application.