A Reinforcement Learning-Based Framework for the Generation and Evolution of Adaptation Rules

One of the challenges in self-adaptive systems concerns how to make adaptation to themselves at runtime in response to possible and even unexpected changes from the environment and/or user goals. A feasible solution to this challenge is rule-based adaptation, in which, adaptation decisions are made according to predefined rules that specify what particular actions should be performed to react to different changing events from the environment. Although it has the characteristic of highly- efficient decision making for adaptation, rule-based adaptation has two limitations: 1. no guarantee that those predefined rules will lead to optimal or nearly-optimal adaptation results; 2. weak support to evolve these rules to cope with non-stationary environment and changeable user goals at runtime. In this paper, we propose a reinforcement learning-based framework to the generation and evolution of software adaptation rules. This framework manifests two key capabilities for self-adaptation: 1. the capability of automatically learning adaptation rules from different goal settings at the offline phase; 2. the capability of automatically evolving adaptation rules from real-time information about the environment and user goals at the online phase. The two capabilities are built on the combination of reinforcement learning and case-based reasoning techniques. This framework improves the existing rule-based adaptation from two points: the flexibility of adaptation logic, and the quality of adaptation rules. We evaluate this framework through a case study of an E-commerce web application, which shows that this framework improves both the efficiency and effectiveness of self-adaptation.