A scalable complex event processing framework for combination of SQL-based continuous queries and C/C++ functions

SQL-based languages are widely used for Software-based Complex Event Processing (CEP) systems. This paper proposes a FPGA acceleration framework to compile a SQL-based event processing language, which is based on the ANSI standard proposal to support event processing, into a high-performance CEP engine on FPGAs. Besides the SQL's primitives such as partitioning, windowing, aggregation and pattern matching, the proposed framework allows C/C++ functions to implement complex algorithms required by real-world applications. The CEP engine also scales very efficiently processing multiple streams in parallel by making use of inexpensive block memories. Experimental results show that our proposed CEP engine, which is compiled for a financial analysis application calculating a popular trading benchmark to capture meaningful market trends, achieves 150M events/sec (20Gbps) processing performance for over 16,000 streams.

[1]  Viktor K. Prasanna,et al.  Multi-stream Regular Expression Matching on FPGA , 2011, 2011 International Conference on Reconfigurable Computing and FPGAs.

[2]  David J. DeWitt,et al.  The Niagara Internet Query System , 2001, IEEE Data Eng. Bull..

[3]  Jason Cong,et al.  Platform-Based Behavior-Level and System-Level Synthesis , 2006, 2006 IEEE International SOC Conference.

[4]  Takashi Takenaka,et al.  Dynamic query switching for complex event processing on FPGAs , 2012, 22nd International Conference on Field Programmable Logic and Applications (FPL).

[5]  Jennifer Widom,et al.  STREAM: The Stanford Stream Data Manager , 2003, IEEE Data Eng. Bull..

[6]  Neil Immerman,et al.  Efficient pattern matching over event streams , 2008, SIGMOD Conference.

[7]  Paulo Marques,et al.  A Performance Study of Event Processing Systems , 2009, TPCTC.

[8]  Jürgen Teich,et al.  On-the-fly Composition of FPGA-Based SQL Query Accelerators Using a Partially Reconfigurable Module Library , 2012, 2012 IEEE 20th International Symposium on Field-Programmable Custom Computing Machines.

[9]  Frederick Reiss,et al.  TelegraphCQ: continuous dataflow processing , 2003, SIGMOD '03.

[10]  N. Immerman,et al.  SASE + : An Agile Language for Kleene Closure over Event Streams , 2007 .

[11]  Qiang Chen,et al.  Aurora : a new model and architecture for data stream management ) , 2006 .

[12]  Satnam Singh,et al.  Designing application specific circuits with concurrent C# programs , 2010, Eighth ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMOCODE 2010).

[13]  Michael Stonebraker,et al.  Aurora: a new model and architecture for data stream management , 2003, The VLDB Journal.

[14]  Kazutoshi Wakabayashi,et al.  “All-in-C” Behavioral Synthesis and Verification with CyberWorkBench , 2008 .

[15]  Jason Cong,et al.  FCUDA: Enabling efficient compilation of CUDA kernels onto FPGAs , 2009, 2009 IEEE 7th Symposium on Application Specific Processors.

[16]  Johannes Gehrke,et al.  Cayuga: A General Purpose Event Monitoring System , 2007, CIDR.

[17]  Jason Helge Anderson,et al.  LegUp: high-level synthesis for FPGA-based processor/accelerator systems , 2011, FPGA '11.

[18]  Gustavo Alonso,et al.  Streams on Wires - A Query Compiler for FPGAs , 2009, Proc. VLDB Endow..

[19]  Ying Xing,et al.  The Design of the Borealis Stream Processing Engine , 2005, CIDR.

[20]  Gustavo Alonso,et al.  Real-time pattern matching with FPGAs , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[21]  Bernhard Seeger,et al.  PIPES: a public infrastructure for processing and exploring streams , 2004, SIGMOD '04.

[22]  Philip S. Yu,et al.  SPADE: the system s declarative stream processing engine , 2008, SIGMOD Conference.

[23]  Frederick Reiss,et al.  TelegraphCQ: Continuous Dataflow Processing for an Uncertain World , 2003, CIDR.

[24]  Neil Immerman,et al.  On Supporting Kleene Closure over Event Streams , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[25]  Scott A. Mahlke,et al.  Streamroller:: automatic synthesis of prescribed throughput accelerator pipelines , 2006, Proceedings of the 4th International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS '06).

[26]  Takashi Takenaka,et al.  20Gbps C-Based Complex Event Processing , 2011, 2011 21st International Conference on Field Programmable Logic and Applications.