Distributed data stream processing (DSP) is used to analyze information and raise alarms in business-critical scenarios such as financial fraud-detection, clickstream processing, network security, traffic control, or real-time KPI computations. Processing this information efficiently is very challenging because the nature of continuous streaming sources is varying in nature: often the amount of data and processing changes with time of day and day of week and frequently has unexpected spikes. Thus, the result is that most DSP computations are either over-provisioned, introducing increased cost and wasted energy, or are under-provisioned and, either incur in performance degradation or denial-of-service, or have to resort to load shedding.
We demonstrate Flood, a scalable, elastic DSP engine that addresses these problems. By using a scalable computing model, MapReduce, and adequately monitoring running computations our system is able to decide, in runtime, if there is a lack or a surplus of resources. Flood then acts autonomically by requesting or releasing computing nodes, without losing tuples or redoing computation, at the same time making sure that latency and throughput requirements are guaranteed.
[1]
Rajeev Thakur,et al.
Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems, IOPADS 1999, May 5, 1999, Atlanta, GA, USA
,
1999,
IOPADS.
[2]
Douglas Stott Parker,et al.
Map-reduce-merge: simplified relational data processing on large clusters
,
2007,
SIGMOD '07.
[3]
Ken Yocum,et al.
Ad-hoc data processing in the cloud
,
2008,
Proc. VLDB Endow..
[4]
Sanjay Ghemawat,et al.
MapReduce: Simplified Data Processing on Large Clusters
,
2004,
OSDI.
[5]
Noah Treuhaft,et al.
Cluster I/O with River: making the fast case common
,
1999,
IOPADS '99.
[6]
Joseph M. Hellerstein,et al.
Flux: an adaptive partitioning operator for continuous query systems
,
2003,
Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).