Dynamic Resource-Efficient Scheduling in Data Stream Management Systems Deployed on Computing Clouds

Scheduling streaming applications in Data Stream Management Systems (DSMS) have been investigated for years. As the deployment platform of DSMS migrates from onpremise clusters to elastic computing clouds, new requirements have emerged for the scheduling process to tackle workload fluctuations with heterogeneous cloud resources. Resource-efficient scheduling is to improve cost-efficiency at runtime by dynamically matching the resource demands of streaming applications with the resource availability of computing nodes. In this paper, we model the scheduling problem as a bin-packing variant and propose a heuristic-based algorithm to solve it with minimised inter-node communication. We also present a prototype scheduler named D-Storm, which extends the original Apache Storm framework into a self-adaptive MAPE-K (Monitoring, Analysis, Planning, Execution, Knowledge) architecture and validates the efficacy and efficiency of our scheduling algorithm. The evaluation carried out on real-world applications such as Twitter Sentiment Analysis proves that D-Storm outperforms the existing resource-aware scheduler and the default Storm scheduler in terms of the reduction of inter-node traffic and application latency, as well as resulting in a significant amount of resource savings through task consolidation.