Introducing Weirs: An Abstraction for Next Generation Streaming Workflows

In HPC applications, it is widely understood that in situ systems will play a significant role in next generation systems. The rate that next-generation leadership machines will be able to generate data will exceed the bandwidths of the planned I/O systems, leading to a need for in situ processing of the resulting data to reduce it. There have been a number of techniques proposed for in situ workflow systems; we are concerned here specifically with the needs of publish/subscribe or streaming-based approaches, which allow the in situ system to be composed of completely separate executables running concurrently on the hardware. In this paper we introduce an abstract known as a Weir. A weir is a useful mechanism to control state in a point to point streaming workflow in the absence of a third party publish subscribe system.

[1]  Karsten Schwan,et al.  Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics , 2014, 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[2]  Arie Shoshani,et al.  Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks , 2014, Concurr. Comput. Pract. Exp..