Shared data-aware dynamic resource provisioning and task scheduling for data intensive applications on hybrid clouds using Aneka

Abstract In the recent years, data-intensive applications have been growing at an increasing rate and there is a critical need to solve the high-performance and scalability issues. Hybrid Cloud Computing paradigm provides a promising solution to harness local infrastructure and remote resources and provide high Quality of Service (QoS) for time sensitive and data-intensive applications. Generally, hybrid cloud deployments have a heterogeneous pool of resources and it becomes a challenging task to efficiently utilize resources to provide optimum results. In modern data hungry applications, it is crucial to optimize bandwidth consumption, latency and networking overheads. Moreover, most of them have large extent of file sharing capability. The existing algorithms do not explicitly consider file sharing scenarios that leads large data transmission times and has severe effects on latency. In this direction, this paper focuses on building upon existing dynamic resource provisioning and task scheduling algorithms to provide better QoS in hybrid cloud environments for data intensive applications in a shared file task environment. The efficiency of proposed algorithms is demonstrated by deploying them on Microsoft Azure using Aneka, a platform for developing scalable applications on the Cloud. Experiments using real-world applications and datasets show that proposed algorithms are able to allocate tasks and extend to public cloud resources more efficiently, reducing deadline violations and improving response times to give response time reduction of upto 40.12% for a sample local alignment search application on genome sequences.