On efficient resource use for scientific workflows in clouds

Abstract The abundance of cloud resources has enabled not only web applications, but also scientific applications to easily scale to meet their objectives, such as performance and costs. However, due to the complex and large-scale nature of scientific workflows, the decision on such scaling (resource management) is much complicated often resulting in inefficient use of resources. In this paper, we present RDAS+ as a resource demand aware scheduling algorithm to optimize resource efficiency for the execution of scientific workflows in clouds. RDAS+ maximizes resource utilization by allocating the minimum number of resources (virtual machines or VMs in clouds) with little sacrifice of completion time (makespan). This optimization eventually leads to cost efficiency for pay-per-use cloud resources. RDAS+ consists of partitioning, resource allocation and task scheduling steps to realize such optimization. We have evaluated RDAS+ using five types of real-world scientific workflows in comparison with three existing algorithms. Our experimental results confirm our claims on achieving resource efficiency. In particular, the average rate of cost savings (32%) outweighs makespan increase (11%). Although these two performance metrics are incompatible, the trade-off RDAS+ optimizes shows significant benefit particularly in clouds.

[1]  Rajkumar Buyya,et al.  A Particle Swarm Optimization-Based Heuristic for Scheduling Workflow Applications in Cloud Computing Environments , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[2]  Sylvain Bouveret,et al.  Characterizing conflicts in fair division of indivisible goods using a scale of criteria , 2016, Autonomous Agents and Multi-Agent Systems.

[3]  H. Varian Equity, Envy and Efficiency , 1974 .

[4]  Rajkumar Buyya,et al.  Cost-based scheduling of scientific workflow applications on utility grids , 2005, First International Conference on e-Science and Grid Computing (e-Science'05).

[5]  Young Choon Lee,et al.  Partitioning-Based Workflow Scheduling in Clouds , 2016, 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA).

[6]  Radu Prodan,et al.  Bi-Criteria Scheduling of Scientific Grid Workflows , 2010, IEEE Transactions on Automation Science and Engineering.

[7]  Xiaorong Li,et al.  ScaleStar: Budget Conscious Scheduling Precedence-Constrained Many-task Workflow Applications in Cloud , 2012, 2012 IEEE 26th International Conference on Advanced Information Networking and Applications.

[8]  Rajkumar Buyya,et al.  Deadline Based Resource Provisioningand Scheduling Algorithm for Scientific Workflows on Clouds , 2014, IEEE Transactions on Cloud Computing.

[9]  Marios D. Dikaiakos,et al.  Scheduling Workflows with Budget Constraints , 2007, Grid 2007.

[10]  Dick H. J. Epema,et al.  Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service Clouds , 2013, Future Gener. Comput. Syst..

[11]  Susanne Albers,et al.  Average-case analyses of first fit and random fit bin packing , 2000, SODA '98.

[12]  Rizos Sakellariou,et al.  A hybrid heuristic for DAG scheduling on heterogeneous systems , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[13]  Radu Prodan,et al.  MOHEFT: A multi-objective list-based method for workflow scheduling , 2012, 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings.

[14]  Chase Qishi Wu,et al.  End-to-End Delay Minimization for Scientific Workflows in Clouds under Budget Constraint , 2015, IEEE Transactions on Cloud Computing.

[15]  D. Simchi-Levi New worst‐case results for the bin‐packing problem , 1994 .

[16]  Marian Bubak,et al.  Cost Optimization of Execution of Multi-level Deadline-Constrained Scientific Workflows on Clouds , 2013, PPAM.

[17]  Rizos Sakellariou,et al.  Budget-Deadline Constrained Workflow Planning for Admission Control , 2013, Journal of Grid Computing.

[18]  Hamid Arabnejad,et al.  A Budget Constrained Scheduling Algorithm for Workflow Applications , 2014, Journal of Grid Computing.

[19]  Bernard Mans,et al.  Resource demand aware scheduling for workflows in clouds , 2017, 2017 IEEE 16th International Symposium on Network Computing and Applications (NCA).

[20]  Ewa Deelman,et al.  Scientific workflows and clouds , 2010, ACM Crossroads.

[21]  Salim Hariri,et al.  Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing , 2002, IEEE Trans. Parallel Distributed Syst..