Genetic-based Multi-criteria Workflow Scheduling with Dynamic Resource Provisioning in Hybrid Large Scale Distributed Systems

Abstract Nowadays, scientific progress in multiple disciplines gave rise to conducting large scale scientific experiments and applications known as High Performance Computing (HPC). These HPC applications are commonly structured as workflows of heavy tasks with large data size and intricate dependencies which are typically performed in large-scale distributed systems (LSDS) such as clusters, Grids and recently Cloud infrastructures. In fact, workflow scheduling in distributed systems, especially in Clouds, is proved to be an NP hard problem. Our main target in this paper is to design a workflow scheduling approach based on the Non-dominated Sorting Genetic Algorithm version 2 (NSGA-II) in hybrid distributed systems by optimizing the Makespan and cost. In this work, we also studied the improvement of the Makespan-Cost trade-off with the scalability concept in the Cloud with our designed approach. For that, we proposed different scenarios dealing with the provisioning strategy of processing nodes alongside an existing resource pool. Conducted experiments show the advantage of Cloud infrastructures against other distributed systems and allow investigating the different factors in correlation with resources provisioning process.