A budget and deadline aware scientific workflow resource provisioning and scheduling mechanism for cloud

Currently in large-scale scientific experiments, scientists often submit scientific workflow jobs at different time. From the view of system, the entire workload is a stream of jobs submitted at an unpredictable time and different job has different priority and deadline. Moreover the cost of performing these jobs cannot exceed a certain budget constraint. Therefore how to perform scientific workflow applications efficiently in cloud has become the urgent problem. However most of existing work didn't consider unpredictable submission time of jobs, as well as budget and deadline constrains. In this paper, we design an elastic resource provisioning and task scheduling mechanism to perform scientific workflows in cloud. Our goal is to complete as many high-priority workflows as possible under budget and deadline constrains. This mechanism consists of three phases: workflow preprocessing, elastic resource provisioning and task scheduling. We perform evaluation with real AMS experiment scientific computing data under different budget constraints. We also consider inaccurate task execution time, VM provisioning delays and task failures in evaluation. The results show that our mechanism achieves a better performance than these reference mechanisms. In addition, the inaccurate task execution time, VM provisioning delays, and task failures do not bring significant impact to mechanism's performance.

[1]  Marty Humphrey,et al.  Scaling and Scheduling to Maximize Application Performance within Budget Constraints in Cloud Workflows , 2013, 2013 IEEE 27th International Symposium on Parallel and Distributed Processing.

[2]  Dick H. J. Epema,et al.  Cost-Driven Scheduling of Grid Workflows Using Partial Critical Paths , 2012 .

[3]  Dick H. J. Epema,et al.  Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service Clouds , 2013, Future Gener. Comput. Syst..

[4]  Marios D. Dikaiakos,et al.  Scheduling Workflows with Budget Constraints , 2007, Grid 2007.

[5]  Dick H. J. Epema,et al.  Cost-driven scheduling of grid workflows using Partial Critical Paths , 2010, 2010 11th IEEE/ACM International Conference on Grid Computing.

[6]  Jarek Nabrzyski,et al.  Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.

[7]  Rajkumar Buyya,et al.  Cost-based scheduling of scientific workflow applications on utility grids , 2005, First International Conference on e-Science and Grid Computing (e-Science'05).

[8]  Jin-Soo Kim,et al.  Cost optimized provisioning of elastic resources for application workflows , 2011, Future Gener. Comput. Syst..

[9]  Francine Berman,et al.  New Grid Scheduling and Rescheduling Methods in the GrADS Project , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[10]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[11]  Marty Humphrey,et al.  Auto-scaling to minimize cost and meet application deadlines in cloud workflows , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).