Learned Autoscaling for Cloud Microservices with Multi-Armed Bandits

As cloud applications shift from monolithic architectures to loosely coupled microservices, several challenges in resource management arise. Application developers are tasked with determining compute capacity needed for each microservice in an application. This allocation dictates both the cost and performance of the application and typically relies on using either machine utilization (e.g. CPU, RAM) metrics. While utilization based policies are often simple to configure, easily understood, and require no training or retraining cost such policies offer no guarantees or expectations of end user latency. We design, implement and evaluate a microservice autoscaling system, COLA, which efficiently learns to manage cluster resources based on user provided end-to-end latency targets and cost objectives rather than optimizing utilization metrics. Our approach, COLA, relies on training a contextual multi armed bandit on representative workloads for an application and uses techniques to generalize performance to unseen workloads. We evaluate workloads of varying complexity including those with a fixed rate, diurnal pattern and dynamic request distribution. Across a set of five open-source microservice applications, we compare COLA against a variety of utilization and machine learning baselines. We find COLA provides the most cost effective autoscaling solution for a desired median or tail latency target on 13 of 18 workloads. On average, clusters managed by COLA cost 25.1% fewer dollars than the next closest alternative that meets a specified target latency. We discuss several optimizations, inspired by systems and machine learning literature, we make during training to efficiently explore the space of possible microservice configurations. These optimizations enable us to train our models over the course of a few hours. The cost savings from managing a cluster with COLA result in the system paying for its own training cost within a few days.