Minimizing financial cost of scientific workflows under deadline constraints in multi-cloud environments

In recent years, cloud platforms have been rapidly developed and deployed around the globe and many large-scale scientific workflows have been migrated to multiple clouds for cost-effective data analysis. In such cloud-based workflow applications, financial cost is a major concern in addition to traditional performance requirements such as execution time. In this paper, we formulate a workflow mapping problem to minimize the financial cost of deadline-constrained scientific workflows executed in multi-cloud environments, referred to as MinCost-MC, which is shown to be NP-complete. Within a generic three-layer workflow execution framework, we propose a Workflow Mapping algorithm for Financial Cost Optimization, referred to as WMFCO. This algorithm takes in consideration storage requirements, I/O operations, and data transfers to minimize the financial cost of a given workflow within a specified deadline. Extensive simulation results show that WMFCO exhibits a superior performance over existing algorithms in terms of financial cost in multi-cloud environments.

[1]  Xiao Liu,et al.  A market-oriented hierarchical scheduling strategy in cloud workflow systems , 2011, The Journal of Supercomputing.

[2]  Rajkumar Buyya,et al.  Deadline Based Resource Provisioningand Scheduling Algorithm for Scientific Workflows on Clouds , 2014, IEEE Transactions on Cloud Computing.

[3]  Marios D. Dikaiakos,et al.  Scheduling Workflows with Budget Constraints , 2007, Grid 2007.

[4]  Mahmoud Naghibzadeh,et al.  Deadline-constrained workflow scheduling in software as a service Cloud , 2012, Sci. Iran..

[5]  Jarek Nabrzyski,et al.  Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds , 2015 .

[6]  Rizos Sakellariou,et al.  A hybrid heuristic for DAG scheduling on heterogeneous systems , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[7]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[8]  Mahmoud Naghibzadeh,et al.  Deadline constrained load balancing level based workflow scheduling for cost optimization , 2017, 2017 2nd IEEE International Conference on Computational Intelligence and Applications (ICCIA).

[9]  R. F. Freund,et al.  Dynamic Mapping of a Class of Independent Tasks onto Heterogeneous Computing Systems , 1999, J. Parallel Distributed Comput..

[10]  Sai Peck Lee,et al.  Cost-aware challenges for workflow scheduling approaches in cloud computing environments: Taxonomy and opportunities , 2015, Future Gener. Comput. Syst..

[11]  C. P. Katti,et al.  Cost effective deadline aware scheduling strategy for workflow applications on virtual machines in cloud computing , 2017, J. King Saud Univ. Comput. Inf. Sci..

[12]  Jun Zhang,et al.  An Ant Colony Optimization Approach to a Grid Workflow Scheduling Problem With Various QoS Requirements , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[13]  Rajkumar Buyya,et al.  Cost-based scheduling of scientific workflow applications on utility grids , 2005, First International Conference on e-Science and Grid Computing (e-Science'05).

[14]  Sai Peck Lee,et al.  Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: A review, classifications, and open issues , 2016, J. Syst. Softw..

[15]  R. Buyya,et al.  A budget constrained scheduling of workflow applications on utility Grids using genetic algorithms , 2006, 2006 Workshop on Workflows in Support of Large-Scale Science.

[16]  Johan Montagnat,et al.  Scientific workflows: Past, present and future , 2017, Future Gener. Comput. Syst..

[17]  Marty Humphrey,et al.  Auto-scaling to minimize cost and meet application deadlines in cloud workflows , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[18]  Albert Y. Zomaya,et al.  Executing Large Scale Scientific Workflow Ensembles in Public Clouds , 2015, 2015 44th International Conference on Parallel Processing.

[19]  Rajkumar Buyya,et al.  Scheduling scientific workflow applications with deadline and budget constraints using genetic algorithms , 2006, Sci. Program..

[20]  Chase Qishi Wu,et al.  Optimizing the Performance of Big Data Workflows in Multi-cloud Environments Under Budget Constraint , 2016, 2016 IEEE International Conference on Services Computing (SCC).

[21]  Rajkumar Buyya,et al.  Multi-objective planning for workflow execution on Grids , 2007, 2007 8th IEEE/ACM International Conference on Grid Computing.

[22]  Thomas J. Hacker,et al.  Flexible resource allocation for reliable virtual cluster computing systems , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[23]  Jin-Soo Kim,et al.  Cost optimized provisioning of elastic resources for application workflows , 2011, Future Gener. Comput. Syst..

[24]  Rajkumar Buyya,et al.  A Dynamic Critical Path Algorithm for Scheduling Scientific Workflow Applications on Global Grids , 2007, Third IEEE International Conference on e-Science and Grid Computing (e-Science 2007).

[25]  Jarek Nabrzyski,et al.  Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds , 2012, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis.