A Scalable Circular Pipeline Design for Multi-Way Stream Joins in Hardware

Efficient real-time analytics are an integral part of a growing number of data management applications such as computational targeted advertising, algorithmic trading, and Internet of Things. In this paper, we primarily focus on accelerating stream joins, arguably one of the most commonly used and resource-intensive operators in stream processing. We propose a scalable circular pipeline design (Circular-MJ) in hardware to orchestrate multi-way join while minimizing data flow disruption. In this circular design, each new tuple (given its origin stream) starts its processing from a specific join core and passes through all respective join cores in a pipeline sequence to produce final results. We further present a novel two-stage pipeline stream join (Stashed-MJ) that uses a best-effort buffering technique (stash) to maintain intermediate results. In a case that an overwrite is detected in the stash, our design automatically resorts to recomputing intermediate results. Our experimental results demonstrate a linear throughput scaling with respect to the number of execution units in hardware.

[1]  Milos Nikolic,et al.  DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views , 2012, Proc. VLDB Endow..

[2]  Jens Teubner,et al.  How soccer players would do stream joins , 2011, SIGMOD '11.

[3]  Philip S. Yu,et al.  CellJoin: a parallel stream join operator for the cell processor , 2009, The VLDB Journal.

[4]  Kaiwen Zhang,et al.  Hardware Acceleration Landscape for Distributed Real-Time Analytics: Virtues and Limitations , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[5]  Hans-Arno Jacobsen,et al.  Multi-query Stream Processing on FPGAs , 2012, 2012 IEEE 28th International Conference on Data Engineering.

[6]  Marina Papatriantafilou,et al.  Scalejoin: A deterministic, disjoint-parallel and skew-resilient stream join , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[7]  Hideyuki Kawashima,et al.  Design and Implementation of a Handshake Join Architecture on FPGA , 2012, IEICE Trans. Inf. Syst..

[8]  Beng Chin Ooi,et al.  Scalable Distributed Stream Join Processing , 2015, SIGMOD Conference.

[9]  Hans-Arno Jacobsen,et al.  Configurable hardware-based streaming architecture using Online Programmable-Blocks , 2015, 2015 IEEE 31st International Conference on Data Engineering.

[10]  Hans-Arno Jacobsen,et al.  SplitJoin: A Scalable, Low-latency Stream Join Architecture with Adjustable Ordering Precision , 2016, USENIX Annual Technical Conference.

[11]  Christoph Koch,et al.  Incremental query evaluation in a ring of databases , 2010, PODS.

[12]  Dionisios N. Pnevmatikatos,et al.  An FPGA-based high-throughput stream join architecture , 2016, 2016 26th International Conference on Field Programmable Logic and Applications (FPL).

[13]  Jeffrey F. Naughton,et al.  Evaluating window joins over unbounded streams , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).