We propose a prototype incremental data migration mechanism for stateful distributed data-parallel dataflow engines with latency objectives. When compared to existing scaling mechanisms, our prototype has the following differentiating characteristics: (i) the mechanism provides tunable granularity for avoiding latency spikes, (ii) reconfigurations can be prepared ahead of time to avoid runtime coordination, and (iii) the implementation only relies on existing dataflow APIs and need not require system modifications. We demonstrate our proposal on example computations with varying amounts of state that needs to be migrated, which is a non-trivial task for systems like Dhalion and Flink. Our implementation, prototyped on Timely Dataflow, provides a scalable stateful operator template compatible with existing APIs that carefully reorganizes data to minimize migration overhead. Compared to naïve approaches we reduce service latencies by orders of magnitude.
[1]
Sriram Rao,et al.
Dhalion: Self-Regulating Stream Processing in Heron
,
2017,
Proc. VLDB Endow..
[2]
M. Abadi,et al.
Naiad: a timely dataflow system
,
2013,
SOSP.
[3]
Daniel Mills,et al.
MillWheel: Fault-Tolerant Stream Processing at Internet Scale
,
2013,
Proc. VLDB Endow..
[4]
Raul Castro Fernandez,et al.
Integrating scale out and fault tolerance in stream processing using operator state management
,
2013,
SIGMOD '13.
[5]
Seif Haridi,et al.
Apache Flink™: Stream and Batch Processing in a Single Engine
,
2015,
IEEE Data Eng. Bull..