Automatic Management of Cloud Applications with Use of Proximal Policy Optimization

Reinforcement learning is a very active field of research with many practical applications. Success in many cases is driven by combining it with Deep Learning. In this paper we present the results of our attempt to use modern advancements in this area for automated management of resources used to host distributed software. We describe the use of an autonomous agent that employs a policy trained with use of Proximal Policy Optimization algorithm. The agent is managing a cloud infrastructure used to process a sample workload. We present the design and architecture of a complete autonomous management system and explain how the management policy was trained. Finally, we compare the performance to the traditional automatic management approach exploited in AWS stack and discuss feasibility to use the presented approach in other scenarios.

[1]  Pawel Koperek,et al.  Evaluating the Use of Policy Gradient Optimization Approach for Automatic Cloud Resource Provisioning , 2019, PPAM.

[2]  Rajkumar Buyya,et al.  Containers Orchestration with Cost-Efficient Autoscaling in Cloud Computing Environments , 2018, ArXiv.

[3]  Rajkumar Buyya,et al.  CloudAnalyst: A CloudSim-Based Visual Modeller for Analysing Cloud Computing Environments and Applications , 2010, 2010 24th IEEE International Conference on Advanced Information Networking and Applications.

[4]  Ivan Porres,et al.  CRAMP: Cost-efficient Resource Allocation for Multiple web applications with Proactive scaling , 2012, 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings.

[5]  Bernd Freisleben,et al.  Distributed Resource Allocation to Virtual Machines via Artificial Neural Networks , 2014, 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing.

[6]  Sergey Levine,et al.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[7]  J. Mościński,et al.  Computer simulation of heuristic reinforcement-learning systems for nuclear power plant load changes control , 1979 .

[8]  Thilo Kielmann,et al.  Autoscaling Web Applications in Heterogeneous Cloud Infrastructures , 2014, 2014 IEEE International Conference on Cloud Engineering.

[9]  David Sinreich,et al.  An architectural blueprint for autonomic computing , 2006 .

[10]  Xin Yao,et al.  A Survey and Taxonomy of Self-Aware and Self-Adaptive Cloud Autoscaling Systems , 2016, ACM Comput. Surv..

[11]  Muhammad Arshad Islam,et al.  Investigation of Cloud Scheduling Algorithms for Resource Utilization Using CloudSim , 2019, Comput. Informatics.

[12]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[13]  Wlodzimierz Funika,et al.  Dynamic Business Metrics-driven Resource Provisioning in Cloud Environments , 2011, PPAM.

[14]  Richard S. Sutton,et al.  Temporal credit assignment in reinforcement learning , 1984 .

[15]  Sanjay P. Ahuja,et al.  Multi-Factor Performance Comparison of Amazon Web Services Elastic Compute Cluster and Google Cloud Platform Compute Engine , 2020, Int. J. Cloud Appl. Comput..

[16]  Calton Pu,et al.  SmartSLA: Cost-Sensitive Management of Virtualized Resources for CPU-Bound Database Services , 2015, IEEE Transactions on Parallel and Distributed Systems.

[17]  Mário M. Freire,et al.  CloudSim Plus: A cloud computing simulation framework pursuing software engineering principles for improved modularity, extensibility and correctness , 2017, 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM).

[18]  Wlodzimierz Funika,et al.  Towards Autonomic Semantic-Based Management of Distributed Applications , 2010, Comput. Sci..

[19]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[20]  Tim Oates,et al.  Automated Cloud Provisioning on AWS using Deep Reinforcement Learning , 2017, ArXiv.

[21]  Wojciech Rzasa,et al.  Predicting Performance in a PaaS Environment: a Case Study for a Web Application , 2017, Comput. Sci..

[22]  Mary Shaw,et al.  Engineering Self-Adaptive Systems through Feedback Loops , 2009, Software Engineering for Self-Adaptive Systems.

[23]  Soonwook Hwang,et al.  An allocation and provisioning model of science cloud for high throughput computing applications , 2013, CAC.

[24]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[25]  Wlodzimierz Funika,et al.  Co-evolution of Fitness Predictors and Deep Neural Networks , 2017, PPAM.

[26]  Rajkumar Buyya,et al.  A reliable and cost-efficient auto-scaling system for web applications using heterogeneous spot instances , 2015, Journal of Network and Computer Applications.

[27]  Cheng-Zhong Xu,et al.  URL: A unified reinforcement learning approach for autonomic cloud management , 2012, J. Parallel Distributed Comput..

[28]  Samuel Kounev,et al.  Model-based self-adaptive resource allocation in virtualized environments , 2011, SEAMS '11.

[29]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[30]  Srinivas Devadas,et al.  Seec: a framework for self-aware management of goals and constraints in computing systems (power-aware computing, accuracy-aware computing, adaptive computing, autonomic computing) , 2013 .

[31]  Fabio Panzieri,et al.  QoS–Aware Clouds , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[32]  Yves Le Traon,et al.  Generic cloud platform multi-objective optimization leveraging models@run.time , 2014, SAC.

[33]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[34]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.