A practical framework for adaptive metaheuristics

Local search methods are useful tools for tackling hard problems such as many combinatorial optimization problems (COP). Experience from mathematics has shown that exploiting regularities in problem solving is beneficial. Consequently, identifying and exploiting regularities in the context of local search methods is deemed to be desirable, too. Due to the complexity of the COPs tackled, regularities might better be detected and learned automatically. This can be achieved by means of machine learning techniques to extend existing local search methods. Learning requires feedback, but in the context of local search methods, instructive feedback is not available. Instead, evaluative feedback can be derived from the cost function of COPs evaluating single solutions, for example. Reinforcement learning (RL) is a machine learning technique that only needs evaluative feedback. A particular RL method is called Q-learning. The present thesis attempts to develop learning local search methods in a general and practical manner. One possibility to enhance local search methods with learning capabilities is by using RL methods. The direct application of existing RL techniques for extending existing local search methods is enabled by the concept of a local search agent (LSA). The advancement of a trajectory-based local search method can be regarded as the interaction of a virtual agent whose states basically consist of solutions and whose actions are composed of arbitrary hierarchical compositions of local search operators. The resulting LSA using RL can then be called a learning LSA. The changes in cost for each move of a learning LSA can be used as reward. Based on these, returns can be computed such that maximizing the return reflects the goal of finding a global or a very good local optimum. The hierarchical structure of LSA actions allows to use so-called ILS-actions. ILS-actions coincide with the application of one iteration of the well-known Iterated Local Search (ILS) metaheuristic. The advantage of this metaheuristic and this kind of action is that only solutions from the subset of local optima -- which must contain any acceptable solution -- are considered and thus introduces a search space abstraction which in turn can improve performance. A learning LSA that employs ILS-actions iteratively will visit local optima in a guided and adaptive manner. The resulting theoretical framework is called Guided Adaptive Iterated Local Search (GAILS). In order to evaluate randomized GAILS algorithms, empirical experiments have to be conducted. Each GAILS algorithm thereby consists of three, mainly independent parts. The first part comprises the actions of a learning LSA which are specific to a problem type. The LSA actions being arbitrary hierarchical compositions of local search operators are implemented through basic local search operators. The second part represents the RL techniques used, which in turn transparently use actions and hence are problem type independent. The third part consists of the function approximators used by RL techniques. The function approximators only require as input a vector of real-valued features and this way are independent from the first two parts. Empirical experiments can be supported by providing a framework that can decouple these three main parts in any GAILS algorithm program instantiation, thus allowing for an arbitrary reuse and combination enabling rapid prototyping. The GAILS implementation framework is such an application framework which is designed to rapidly implement learning LSAs reflecting the separation of a learning LSA into its three main parts. It provides generic interfaces between components of the three parts and this way provides for a separation of problem type specific states from search control. It also provides for a separation of search control from the state of the search control unit. Hierarchically built actions are mapped to object hierarchies. Two GAILS algorithms according to Q-learning algorithm Q(0) and Q(λ) that are based on ILS-actions were developed, built and compared to corresponding standard implementations of the ILS metaheuristic. These so-called Q-ILS algorithms were tested for two problems using different function approximators. The results showed that learning useful policies and transfer of what was learned across multiple problem instances, even of different sizes, is possible and useful.