RADAR: Adaptive Rate Allocation in Distributed Stream Processing Systems under Bursty Workloads

In the recent years we have witnessed a proliferation of distributed stream processing systems that need to operate under bursty workloads. Examples include road traffic control, processing of financial feeds, network monitoring and real-time sensor data analysis systems. Meeting the QoS requirements of the stream processing systems under burstiness is a challenging process. In this paper we present our approach for adaptive rate allocation within the distributed stream processing system to meet the end-to-end execution time and rate demands of the applications. Our algorithm determines the rates of the application components at runtime, with respect to the QoS constraints, to compensate for delays experienced by the components or to react to sudden bursts of load. Our technique is distributed and low-cost. Our detailed experimental results over our Synergy middleware illustrate that our approach is practical, depicts good performance and has low resource overhead.

[1]  Navendu Jain,et al.  Adaptive Control of Extreme-scale Stream Processing Systems , 2006, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06).

[2]  Philip S. Yu,et al.  Optimal Component Composition for Scalable Stream Processing , 2005, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05).

[3]  Steven H. Low,et al.  Optimization flow control—I: basic algorithm and convergence , 1999, TNET.

[4]  Jiannong Cao,et al.  User Density Sensitive P2P Streaming in Wireless Mesh Networks , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[5]  Vana Kalogeraki,et al.  Accommodating bursts in distributed stream processing systems , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[6]  Cristian Lumezanu,et al.  Online Optimization for Latency Assignment in Distributed Real-Time Systems , 2008, 2008 The 28th International Conference on Distributed Computing Systems.

[7]  Douglas C. Schmidt,et al.  Toward Effective Multi-Capacity Resource Allocation in Distributed Real-Time and Embedded Systems , 2008, 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC).

[8]  Daniel Pérez Palomar,et al.  A tutorial on decomposition methods for network utility maximization , 2006, IEEE Journal on Selected Areas in Communications.

[9]  Chenyang Lu,et al.  Feedback utilization control in distributed real-time systems with end-to-end tasks , 2005, IEEE Transactions on Parallel and Distributed Systems.

[10]  Sang Hyuk Son,et al.  Prediction-Based QoS Management for Real-Time Data Streams , 2006, 2006 27th IEEE International Real-Time Systems Symposium (RTSS'06).

[11]  Stanley B. Zdonik,et al.  Staying FIT: Efficient Load Shedding Techniques for Distributed Stream Processing , 2007, VLDB.

[12]  Ying Xing,et al.  Providing resiliency to load variations in distributed stream processing , 2006, VLDB.

[13]  Xiaohui Gu,et al.  Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems , 2006, Middleware.

[14]  Gruia-Catalin Roman,et al.  Real-Time Query Scheduling for Wireless Sensor Networks , 2007, RTSS 2007.

[15]  Rene L. Cruz,et al.  A calculus for network delay, Part I: Network elements in isolation , 1991, IEEE Trans. Inf. Theory.

[16]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[17]  Evgenia Smirni,et al.  AWAIT: Efficient Overload Management for Busy Multi-tier Web Services under Bursty Workloads , 2010, ICWE.

[18]  Chenyang Lu,et al.  Optimal Discrete Rate Adaptation for Distributed Real-Time Systems , 2007, 28th IEEE International Real-Time Systems Symposium (RTSS 2007).

[19]  Alexandre M. Bayen,et al.  Evaluation of traffic data obtained via GPS-enabled mobile phones: The Mobile Century field experiment , 2009 .

[20]  Rene L. Cruz.  A calculus for network delay. I - Network elements in isolation. II - Network analysis , 1991 .

[21]  Donald F. Towsley,et al.  Distributed Resource Management and Admission Control of Stream Processing Systems with Max Utility , 2007, 27th International Conference on Distributed Computing Systems (ICDCS '07).

[22]  Giorgio C. Buttazzo,et al.  Resource Reservation in Dynamic Real-Time Systems , 2004, Real-Time Systems.