R-Storm: Resource-Aware Scheduling in Storm

The era of big data has led to the emergence of new systems for real-time distributed stream processing, e.g., Apache Storm is one of the most popular stream processing systems in industry today. However, Storm, like many other stream processing systems lacks an intelligent scheduling mechanism. The default round-robin scheduling currently deployed in Storm disregards resource demands and availability, and can therefore be inefficient at times. We present R-Storm (Resource-Aware Storm), a system that implements resource-aware scheduling within Storm. R-Storm is designed to increase overall throughput by maximizing resource utilization while minimizing network latency. When scheduling tasks, R-Storm can satisfy both soft and hard resource constraints as well as minimizing network distance between components that communicate with each other. We evaluate R-Storm on set of micro-benchmark Storm applications as well as Storm applications used in production at Yahoo! Inc. From our experimental results we conclude that R-Storm achieves 30-47% higher throughput and 69-350% better CPU utilization than default Storm for the micro-benchmarks. For the Yahoo! Storm applications, R-Storm outperforms default Storm by around 50% based on overall throughput. We also demonstrate that R-Storm performs much better when scheduling multiple Storm applications than default Storm.

[1]  Yuguang Fang,et al.  Multiple multidimensional knapsack problem and its applications in cognitive radio networks , 2008, MILCOM 2008 - 2008 IEEE Military Communications Conference.

[2]  S. Martello,et al.  Dynamic Programming and Strong Bounds for the 0-1 Knapsack Problem , 1999 .

[3]  P. P. Chakrabarti,et al.  Reducing Reexpansions in Iterative-Deepening Search by Controlling Cutoff Bounds , 1991, Artif. Intell..

[4]  Sanjeev Khanna,et al.  A Polynomial Time Approximation Scheme for the Multiple Knapsack Problem , 2005, SIAM J. Comput..

[5]  Jordi Torres,et al.  Resource-Aware Adaptive Scheduling for MapReduce Clusters , 2011, Middleware.

[6]  P. Hammer,et al.  Quadratic knapsack problems , 1980 .

[7]  H. Greenberg,et al.  A Branch Search Algorithm for the Knapsack Problem , 1970 .

[8]  Sanjay V. Rajopadhye,et al.  Unbounded knapsack problem: Dynamic programming revisited , 2000, Eur. J. Oper. Res..

[9]  Sanjeev Khanna,et al.  A PTAS for the multiple knapsack problem , 2000, SODA '00.

[10]  Jiming Liu,et al.  A Mini-Swarm for the Quadratic Knapsack Problem , 2007, 2007 IEEE Swarm Intelligence Symposium.

[11]  Mohammad Hosseini,et al.  Energy-Efficient 3D Texture Streaming for Mobile Games , 2014, MoVid@MMSys.

[12]  Mohammad Hosseini,et al.  Energy-budget-compliant adaptive 3D texture streaming in mobile games , 2013, MMSys.

[13]  Jignesh M. Patel,et al.  Twitter Heron: Stream Processing at Scale , 2015, SIGMOD Conference.

[14]  Vassilis Zissimopoulos,et al.  An approximation algorithm for solving unconstrained two-dimensional knapsack problems , 1995 .

[15]  Reuven Cohen,et al.  Multi-dimensional OFDMA scheduling in a wireless network with relay nodes , 2014, INFOCOM.

[16]  Kun-Lung Wu,et al.  SODA: An Optimizing Scheduler for Large-Scale Stream-Based Distributed Computer Systems , 2008, Middleware.

[17]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[18]  Mohammad Hosseini,et al.  Coordinated bandwidth adaptations for distributed 3D tele-immersive systems , 2015, MMVE@MMSys.

[19]  Musa A. Mammadov,et al.  Global Optimality Conditions and Optimization Methods for Quadratic Knapsack Problems , 2011, J. Optim. Theory Appl..

[20]  Roberto Baldoni,et al.  Adaptive online scheduling in storm , 2013, DEBS.