Adaptive Structural Hyper-Parameter Configuration by Q-Learning

Tuning hyper-parameters for evolutionary algorithms is an important issue in computational intelligence. Performance of an evolutionary algorithm depends not only on its operation strategy design, but also on its hyper-parameters. Hyper-parameters can be categorized in two dimensions as structural/numerical and time-invariant/time-variant. Particularly, structural hyper-parameters in existing studies are usually tuned in advance for time-invariant parameters, or with hand-crafted scheduling for time-invariant parameters. In this paper, we make the first attempt to model the tuning of structural hyper-parameters as a reinforcement learning problem, and present to tune the structural hyper-parameter which controls computational resource allocation in the CEC 2018 winner algorithm by Q-learning. Experimental results show favorably against the winner algorithm on the CEC 2018 test functions.

[1]  Alexander Mendiburu,et al.  Bayesian optimization for parameter tuning in evolutionary algorithms , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[2]  Qingfu Zhang,et al.  DE/EDA: A new evolutionary algorithm for global optimization , 2005, Inf. Sci..

[3]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[4]  Xin Yao,et al.  Automatic Parameter Tuning using Bayesian Optimization Method , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[5]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[6]  Yuhui Shi,et al.  Hybrid Sampling Evolution Strategy for Solving Single Objective Bound Constrained Problems , 2018, 2018 IEEE Congress on Evolutionary Computation (CEC).

[7]  Irene Moser,et al.  A Systematic Literature Review of Adaptive Parameter Control Methods for Evolutionary Algorithms , 2016, ACM Comput. Surv..

[8]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[9]  Xin Yao,et al.  A Survey of Automatic Parameter Tuning Methods for Metaheuristics , 2020, IEEE Transactions on Evolutionary Computation.

[10]  P. N. Suganthan,et al.  Differential Evolution Algorithm With Strategy Adaptation for Global Numerical Optimization , 2009, IEEE Transactions on Evolutionary Computation.

[11]  Janez Brest,et al.  Single objective real-parameter optimization: Algorithm jSO , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[12]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[13]  P. N. Suganthan,et al.  Differential Evolution: A Survey of the State-of-the-Art , 2011, IEEE Transactions on Evolutionary Computation.

[14]  Kenneth V. Price,et al.  An introduction to differential evolution , 1999 .

[15]  Arthur C. Sanderson,et al.  JADE: Adaptive Differential Evolution With Optional External Archive , 2009, IEEE Transactions on Evolutionary Computation.

[16]  Jianyong Sun,et al.  Two-stage EDA-based approach for all optical WDM mesh network survivability under SRLG constraints , 2011, Appl. Soft Comput..

[17]  Peter I. Frazier,et al.  A Tutorial on Bayesian Optimization , 2018, ArXiv.

[18]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[19]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .