A scheduling scheme in the cloud computing environment using deep Q-learning

Abstract Task scheduling, which plays a vital role in cloud computing, is a critical factor that determines the performance of cloud computing. From the booming economy of information processing to the increasing need of quality of service (QoS) in the business of networking, the dynamic task-scheduling problem has attracted worldwide attention. Due to its complexity, task scheduling has been defined and classified as an NP-hard problem. Additionally, most dynamic online task scheduling often manages tasks in a complex environment, which makes it even more challenging to balance and satisfy the benefits of each aspect of cloud computing. In this paper, we propose a novel artificial intelligence algorithm, called deep Q-learning task scheduling (DQTS), that combines the advantages of the Q-learning algorithm and a deep neural network. This new approach is aimed at solving the problem of handling directed acyclic graph (DAG) tasks in a cloud computing environment. The essential idea of our approach uses the popular deep Q-learning (DQL) method in task scheduling, where fundamental model learning is primarily inspired by DQL. Based on developments in WorkflowSim, experiments are conducted that comparatively consider the variance of makespan and load balance in task scheduling. Both simulation and real-life experiments are conducted to verify the efficiency of optimization and learning abilities in DQTS. The result shows that when compared with several standard algorithms precoded in WorkflowSim, DQTS has advantages regarding learning ability, containment, and scalability. In this paper, we have successfully developed a new method for task scheduling in cloud computing.

[1]  Ümit V. Çatalyürek,et al.  Compaction of Schedules and a Two-Stage Approach for Duplication-Based DAG Scheduling , 2009, IEEE Transactions on Parallel and Distributed Systems.

[2]  Shijun Liu,et al.  DRL-Scheduling: An Intelligent QoS-Aware Job Scheduling Framework for Applications in Clouds , 2018, IEEE Access.

[3]  Li Wang,et al.  Hierarchical Deep Multiagent Reinforcement Learning , 2018, ArXiv.

[4]  Hamid Arabnejad,et al.  List Scheduling Algorithm for Heterogeneous Systems by an Optimistic Cost Table , 2014, IEEE Transactions on Parallel and Distributed Systems.

[5]  Sridhar Mahadevan,et al.  Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes , 2007, J. Mach. Learn. Res..

[6]  Junzhou Luo,et al.  Data Placement and Task Scheduling Optimization for Data Intensive Scientific Workflow in Multiple Data Centers Environment , 2014 .

[7]  Zhaohong Wang,et al.  Dynamic Engineering Multi-criteria Decision Making Model Optimized by Entropy Weight for Evaluating Bid , 2012 .

[8]  Rajkumar Buyya,et al.  CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms , 2011, Softw. Pract. Exp..

[9]  Miron Livny,et al.  Pegasus, a workflow management system for science automation , 2015, Future Gener. Comput. Syst..

[10]  Kenli Li,et al.  Scheduling Precedence Constrained Stochastic Tasks on Heterogeneous Cluster Systems , 2015, IEEE Transactions on Computers.

[11]  Ewa Deelman,et al.  WorkflowSim: A toolkit for simulating scientific workflows in distributed environments , 2012, 2012 IEEE 8th International Conference on E-Science.

[12]  Goran Strbac,et al.  Recurrent Deep Multiagent Q-Learning for Autonomous Agents in Future Smart Grid , 2018, AAMAS.

[13]  Luan Teylo,et al.  A hybrid evolutionary algorithm for task scheduling and data assignment of data-intensive scientific workflows on clouds , 2017, Future Gener. Comput. Syst..

[14]  Qingsheng Zhu,et al.  Deadline-Constrained Cost Optimization Approaches for Workflow Scheduling in Clouds , 2017, IEEE Transactions on Parallel and Distributed Systems.

[15]  S. Haykin,et al.  A Q-learning-based dynamic channel assignment technique for mobile communication systems , 1999 .

[16]  Dharma P. Agrawal,et al.  Improving scheduling of tasks in a heterogeneous environment , 2004, IEEE Transactions on Parallel and Distributed Systems.

[17]  Salim Hariri,et al.  Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing , 2002, IEEE Trans. Parallel Distributed Syst..

[18]  Li Fu,et al.  Time and Energy Optimization Algorithms for the Static Scheduling of Multiple Workflows in Heterogeneous Computing System , 2017, Journal of Grid Computing.

[19]  Xin Shen,et al.  Real-time workflows oriented online scheduling in uncertain cloud environment , 2017, The Journal of Supercomputing.

[20]  Mei-Hui Su,et al.  Characterization of scientific workflows , 2008, 2008 Third Workshop on Workflows in Support of Large-Scale Science.

[21]  Hong Liu,et al.  QL-HEFT: a novel machine learning scheduling scheme base on cloud computing environment , 2019, Neural Computing and Applications.

[22]  Ji Li,et al.  DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers , 2018, 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC).

[23]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[24]  Witold Pedrycz,et al.  A Clustering-Based Graph Laplacian Framework for Value Function Approximation in Reinforcement Learning , 2014, IEEE Transactions on Cybernetics.

[25]  Jie Xu,et al.  Data-driven dynamic resource scheduling for network slicing: A Deep reinforcement learning approach , 2019, Inf. Sci..

[26]  Xiaoyong Tang,et al.  Budget‐constraint stochastic task scheduling on heterogeneous cloud systems , 2017, Concurr. Comput. Pract. Exp..

[27]  Seung Ho Hong,et al.  A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach , 2018, Applied Energy.

[28]  Tao Yang,et al.  A Comparison of Clustering Heuristics for Scheduling Directed Acycle Graphs on Multiprocessors , 1992, J. Parallel Distributed Comput..

[29]  Keqin Li,et al.  Heterogeneity-driven end-to-end synchronized scheduling for precedence constrained tasks and messages on networked embedded systems , 2015, J. Parallel Distributed Comput..

[30]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[31]  Jian Ma,et al.  Learning-Based Energy-Efficient Data Collection by Unmanned Vehicles in Smart Cities , 2018, IEEE Transactions on Industrial Informatics.

[32]  Weiwei Lin,et al.  A list scheduling algorithm for heterogeneous systems based on a critical node cost table and pessimistic cost table , 2017, Concurr. Comput. Pract. Exp..

[33]  Ann L. Chervenak,et al.  Characterizing and profiling scientific workflows , 2013, Future Gener. Comput. Syst..

[34]  Ewa Deelman,et al.  Community Resources for Enabling Research in Distributed Scientific Workflows , 2014, 2014 IEEE 10th International Conference on e-Science.

[35]  Jianye Hao,et al.  Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces , 2019, IJCAI.

[36]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[37]  Rizos Sakellariou,et al.  Scheduling multiple DAGs onto heterogeneous systems , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[38]  Laurence T. Yang,et al.  Energy-Efficient Scheduling for Real-Time Systems Based on Deep Q-Learning Model , 2019, IEEE Transactions on Sustainable Computing.

[39]  Gavin Taylor,et al.  Kernelized value function approximation for reinforcement learning , 2009, ICML '09.

[40]  Aihua Liang,et al.  A Novel, Energy-Aware Task Duplication-Based Scheduling Algorithm of Parallel Tasks on Clusters , 2016 .

[41]  Kenli Li,et al.  A novel task scheduling scheme in a cloud computing environment using hybrid biogeography-based optimization , 2019, Soft Comput..

[42]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[43]  Radu Prodan,et al.  Multi-objective workflow scheduling in Amazon EC2 , 2014, Cluster Computing.

[44]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[45]  Hidenori Nakazato,et al.  Prior node selection for scheduling workflows in a heterogeneous system , 2017, J. Parallel Distributed Comput..

[46]  Hamid Arabnejad,et al.  Multi-QoS constrained and Profit-aware scheduling approach for concurrent workflows on heterogeneous systems , 2017, Future Gener. Comput. Syst..

[47]  Eric R. Ziecel.  Four Days With Dr. Deming , 1995 .

[48]  Goran Strbac,et al.  Recurrent Deep Multiagent Q-Learning for Autonomous Brokers in Smart Grid , 2018, IJCAI.

[49]  Kenli Li,et al.  Maximizing reliability with energy conservation for parallel task scheduling in a heterogeneous cluster , 2015, Inf. Sci..