Multi-Query Optimization for Complex Event Processing in SAP ESP

SAP Event Stream Processor (ESP) platform aims at delivering real-time stream processing and analytics in many time-critical areas such as Capital Markets, Internet of Things (IoT) and Data Center Intelligence. SAP ESP allows users to realize complex event processing (CEP) in the form of pattern queries. In this paper, we present MOTTO – a multi-query optimizer in SAP ESP in order to improve the performance of many concurrent pattern queries. This is motivated by the observations that many real-world applications usually have concurrent pattern queries working on the same data streams, leading to tremendous sharing opportunities among queries. In MOTTO, we leverage three major sharing techniques, namely merge, decomposition and operator transformation sharing, to reduce redundant computation among pattern queries. In addition, MOTTO supports nested pattern queries as well as pattern queries with different window sizes. The experiments demonstrate the efficiency of the MOTTO with real-world application scenarios and sensitivity studies.

[1]  Bingsheng He,et al.  Revisiting the Design of Data Stream Processing Systems on Multi-Core Processors , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[2]  Christian Hansen,et al.  What next? , 1994, Nature.

[3]  Philip S. Yu,et al.  Processing high data rate streams in System S , 2011, J. Parallel Distributed Comput..

[4]  Alex Zelikovsky,et al.  A series of approximation algorithms for the acyclic directed steiner tree problem , 1997, Algorithmica.

[5]  Sudipto Guha,et al.  Approximation algorithms for directed Steiner problems , 1999, SODA '98.

[6]  Samuel Madden,et al.  ZStream: a cost-based query processor for adaptively detecting composite events , 2009, SIGMOD Conference.

[7]  Anat Bremler-Barr,et al.  CompactDFA: Scalable Pattern Matching Using Longest Prefix Match Solutions , 2014, IEEE/ACM Transactions on Networking.

[8]  Chetan Gupta,et al.  E-Cube: multi-dimensional event sequence analysis using hierarchical pattern query sharing , 2011, SIGMOD '11.

[9]  Ugur Çetintemel,et al.  Plan-based complex event detection across distributed sources , 2008, Proc. VLDB Endow..

[10]  Elke A. Rundensteiner,et al.  Scalable Pattern Sharing on Event Streams* , 2016, SIGMOD Conference.

[11]  Chetan Gupta,et al.  CHAOS: A Data Stream Analysis Architecture for Enterprise Applications , 2009, 2009 IEEE Conference on Commerce and Enterprise Computing.

[12]  Wang Yongheng,et al.  The Research on Complex Event  Processing Method of Internet of Things , 2013, 2013 Fifth International Conference on Measuring Technology and Mechatronics Automation.

[13]  Chetan Gupta,et al.  High-performance nested CEP query processing over event streams , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[14]  Kun-Lung Wu,et al.  Auto-parallelizing stateful distributed streaming applications , 2012, 2012 21st International Conference on Parallel Architectures and Compilation Techniques (PACT).

[15]  Zhengping Qian,et al.  TimeStream: reliable stream computation in the cloud , 2013, EuroSys '13.

[16]  Van Laarhoven,et al.  Exact and heuristic algorithms for the Euclidean Steiner tree problem , 2010 .

[17]  Johannes Gehrke,et al.  Rule-based multi-query optimization , 2009, EDBT '09.

[18]  M. Crochemore,et al.  On-line construction of suffix trees , 2002 .

[19]  Johannes Gehrke,et al.  Distributed event stream processing with non-deterministic finite automata , 2009, DEBS '09.

[20]  M. Lévěque,et al.  What Is Next , 2014 .

[21]  Abilio Lucena,et al.  A Branch and Cut Algorithm for the Steiner Problem in Graphs , 1998 .

[22]  Peter R. Pietzuch,et al.  Distributed complex event processing with query rewriting , 2009, DEBS '09.