Self-Learning Cloud Controllers: Fuzzy Q-Learning for Knowledge Evolution

Auto-scaling features enable cloud applications to maintain enough resources to satisfy demand spikes, reduce costs and keep performance in check. Most auto-scaling strategies rely on a predefined set of rules to scale up/down the required resources depending on the application usage. Those rules are however difficult to devise and generalize, and users are often left alone tuning auto-scale parameters of essentially blackbox applications. In this paper, we propose a novel fuzzy reinforcement learning controller, FQL4KE, which automatically scales up or down resources to meet performance requirements. The Q-Learning technique, a model-free reinforcement learning strategy, frees users of most tuning parameters. FQL4KE has been successfully applied and we therefore think that a fuzzy controller with Q-Learning is indeed a promising combination for auto-scaling resources.

[1]  Thomas Vogel,et al.  Software Engineering Meets Control Theory , 2015, 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems.

[2]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[3]  Valérie Issarny,et al.  Dynamic decision networks for decision-making in self-adaptive systems: A case study , 2013, 2013 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[4]  Xiaohui Gu,et al.  CloudScale: elastic resource scaling for multi-tenant cloud systems , 2011, SoCC.

[5]  Rizos Sakellariou,et al.  Enacting SLAs in Clouds Using Rules , 2011, Euro-Par.

[6]  Michael I. Jordan,et al.  Statistical Machine Learning Makes Automatic Control Practical for Internet Datacenters , 2009, HotCloud.

[7]  Yudi Wei,et al.  DynaQoS: Model-free self-tuning fuzzy control of virtualized resources for QoS provisioning , 2011, 2011 IEEE Nineteenth IEEE International Workshop on Quality of Service.

[8]  José Antonio Lozano,et al.  A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments , 2014, Journal of Grid Computing.

[9]  Jeffrey S. Chase,et al.  Automated control for elastic storage , 2010, ICAC '10.

[10]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[11]  Danilo Ardagna,et al.  Quality-of-service in cloud computing: modeling techniques and their applications , 2014, Journal of Internet Services and Applications.

[12]  Parijat Dube,et al.  Adaptive, Model-driven Autoscaling for Cloud Applications , 2014, ICAC.

[13]  Guilherme Galante,et al.  A Survey on Cloud Computing Elasticity , 2012, 2012 IEEE Fifth International Conference on Utility and Cloud Computing.

[14]  Calton Pu,et al.  Who Is Your Neighbor: Net I/O Performance Interference in Virtualized Clouds , 2013, IEEE Transactions on Services Computing.

[15]  Sooyong Park,et al.  Reinforcement learning-based dynamic adaptation planning method for architecture-based self-managed software , 2009, 2009 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems.

[16]  John Mylopoulos,et al.  Rationalism with a dose of empiricism: Case-based reasoning for requirements-driven self-adaptation , 2014, 2014 IEEE 22nd International Requirements Engineering Conference (RE).

[17]  Isis Truck,et al.  Using Reinforcement Learning for Autonomic Resource Allocation in Clouds: towards a fully automated workflow , 2011 .

[18]  Enda Barrett,et al.  Applying reinforcement learning towards automating resource allocation and application scalability in the cloud , 2013, Concurr. Comput. Pract. Exp..

[19]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[20]  Jing Xu,et al.  On the Use of Fuzzy Modeling in Virtualized Data Center Management , 2007, Fourth International Conference on Autonomic Computing (ICAC'07).

[21]  Sam Malek,et al.  FUSION: a framework for engineering self-tuning self-adaptive software systems , 2010, FSE '10.

[22]  Claus Pahl,et al.  Cloud Migration Research: A Systematic Review , 2013, IEEE Transactions on Cloud Computing.

[23]  Mohammad Ghafari,et al.  A Framework for Classifying and Comparing Architecture-centric Software Evolution Research , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[24]  Richard N. Taylor,et al.  Using Architectural Models to Manage and Visualize Runtime Adaptation , 2009, Computer.

[25]  Paul Marshall,et al.  Elastic Site: Using Clouds to Elastically Extend Site Resources , 2010, 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing.

[26]  Cheng-Zhong Xu,et al.  Coordinated Self-Configuration of Virtual Machines and Appliances Using a Model-Free Learning Approach , 2013, IEEE Transactions on Parallel and Distributed Systems.

[27]  Eddy Caron,et al.  Auto-Scaling, Load Balancing and Monitoring in Commercial and Open-Source Clouds , 2011 .

[28]  Giovanni Toffetti Carughi,et al.  Kriging-Based Self-Adaptive Cloud Controllers , 2016, IEEE Transactions on Services Computing.

[29]  Rajarshi Das,et al.  A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation , 2006, 2006 IEEE International Conference on Autonomic Computing.

[30]  Ladan Tahvildari,et al.  Adaptive Action Selection in Autonomic Software Using Reinforcement Learning , 2008, Fourth International Conference on Autonomic and Autonomous Systems (ICAS'08).

[31]  P. Glorennec,et al.  Fuzzy Q-learning , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[32]  Helmut Hlavacs,et al.  An Intelligent and Adaptive Threshold-Based Schema for Energy and Performance Efficient Dynamic VM Consolidation , 2013, EE-LSDS.

[33]  Claus Pahl,et al.  Autonomic resource provisioning for cloud-based software , 2014, SEAMS 2014.

[34]  Jesper Andersson,et al.  Knowledge evolution in autonomic software product lines , 2011, SPLC '11.

[35]  Bradley R. Schmerl,et al.  Rainbow: architecture-based self-adaptation with reusable infrastructure , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[36]  David Garlan,et al.  Rainbow: architecture-based self-adaptation with reusable infrastructure , 2004 .

[37]  Giovanni Toffetti Carughi,et al.  Assurance of Self-adaptive Controllers for the Cloud , 2013, Assurances for Self-Adaptive Systems.

[38]  Eunseok Lee,et al.  Model-based reinforcement learning approach for planning in self-adaptive software system , 2015, IMCOM.

[39]  Rajarshi Das,et al.  Utility functions in autonomic systems , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[40]  Jeff Magee,et al.  From goals to components: a combined approach to self-management , 2008, SEAMS '08.

[41]  Xiaobo Zhou,et al.  Autonomic Provisioning with Self-Adaptive Neural Fuzzy Control for Percentile-Based Delay Guarantee , 2013, TAAS.

[42]  Richard N. Taylor,et al.  Policy-based self-adaptive architectures: a feasibility study in the robotics domain , 2008, SEAMS '08.

[43]  Sam Malek,et al.  Ieee Transactions on Software Engineering 1 a Learning-based Framework for Engineering Feature-oriented Self-adaptive Software Systems , 2022 .

[44]  Cheng-Zhong Xu,et al.  URL: A unified reinforcement learning approach for autonomic cloud management , 2012, J. Parallel Distributed Comput..

[45]  Marshall Copeland,et al.  Microsoft Azure , 2015, Apress.

[46]  Richard N. Taylor,et al.  Towards a knowledge-based approach to architectural adaptation management , 2004, WOSS '04.

[47]  Marco Aurélio Stelmar Netto,et al.  Evaluating Auto-scaling Strategies for Cloud Computing Environments , 2014, 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication Systems.

[48]  Mina Sedaghat,et al.  Capacity Management Approaches for Compute Clouds , 2013 .