GraphCEP: real-time data analytics using parallel complex event and graph processing

In recent years, the proliferation of highly dynamic graph-structured data streams fueled the demand for real-time data analytics. For instance, detecting recent trends in social networks enables new applications in areas such as disaster detection, business analytics or health-care. Parallel Complex Event Processing has evolved as the paradigm of choice to analyze data streams in a timely manner, where the incoming data streams are split and processed independently by parallel operator instances. However, the degree of parallelism is limited by the feasibility of splitting the data streams into independent parts such that correctness of event processing is still ensured. In this paper, we overcome this limitation for graph-structured data by further parallelizing individual operator instances using modern graph processing systems. These systems partition the graph data and execute graph algorithms in a highly parallel fashion, for instance using cloud resources. To this end, we propose a novel graph-based Complex Event Processing system GraphCEP and evaluate its performance in the setting of two case studies from the DEBS Grand Challenge 2016.

[1]  Aart J. C. Bik,et al.  Pregel: a system for large-scale graph processing , 2010, SIGMOD Conference.

[2]  Joseph Gonzalez,et al.  PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs , 2012, OSDI.

[3]  Kurt Rothermel,et al.  GrapH: Heterogeneity-Aware Graph Computation with Adaptive Partitioning , 2016, 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS).

[4]  Kurt Rothermel,et al.  MigCEP: operator migration for mobility driven distributed complex event processing , 2013, DEBS.

[5]  Vincenzo Gulisano,et al.  The DEBS 2016 grand challenge , 2016, DEBS.

[6]  Alessandro Margara,et al.  TESLA: a formally defined event specification language , 2010, DEBS '10.

[7]  Kurt Rothermel,et al.  Cordies: expressive event correlation in distributed systems , 2010, DEBS '10.

[8]  Lakshmish Ramaswamy,et al.  Towards efficient query processing on massive time-evolving graphs , 2012, 8th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom).

[9]  David A. Bader,et al.  STINGER: High performance data structure for streaming graphs , 2012, 2012 IEEE Conference on High Performance Extreme Computing.

[10]  Kurt Rothermel,et al.  Moving range queries in distributed complex event processing , 2012, DEBS.

[11]  Nesime Tatbul,et al.  RIP: run-based intra-query parallelism for scalable complex event processing , 2013, DEBS.

[12]  Enhong Chen,et al.  Kineograph: taking the pulse of a fast-changing and connected world , 2012, EuroSys '12.

[13]  Charith Wickramaarachchi,et al.  Enabling Real-time Pro-active Analytics on Streaming Graphs , 2014 .

[14]  Robert Grimm,et al.  A catalog of stream processing optimizations , 2014, ACM Comput. Surv..

[15]  Kurt Rothermel,et al.  Predictable Low-Latency Event Detection With Parallel Complex Event Processing , 2015, IEEE Internet of Things Journal.

[16]  Kurt Rothermel,et al.  Meeting predictable buffer limits in the parallel execution of event processing operators , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[17]  Kurt Rothermel,et al.  Rollback-recovery without checkpoints in distributed event processing systems , 2013, DEBS '13.

[18]  Frederick Reiss,et al.  TelegraphCQ: continuous dataflow processing , 2003, SIGMOD '03.

[19]  Ruben Mayer,et al.  Distributed complex event processing for mobile large-scale video applications , 2014, Middleware.

[20]  Peter R. Pietzuch,et al.  Distributed complex event processing with query rewriting , 2009, DEBS '09.

[21]  Kurt Rothermel,et al.  Hybrid Content-Based Routing Using Network and Application Layer Filtering , 2016, 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS).