Application of Evolutionary Reinforcement Learning (ERL) Approach in Control Domain: A Review

Evolutionary algorithms have come to take a centre stage in diverse areas spanning multiple applications. Reinforcement learning is a novel paradigm that has recently evolved as a major control technique. This paper presents a concise review on implementing reinforcement learning with evolutionary algorithms, e.g. genetic algorithm (GA), particle swarm optimization (PSO), ant colony optimization (ACO), to several benchmark control problems, e.g. inverted pendulum, cart–pole problem, mobile robots. Some techniques have combined Q-Learning with evolutionary approaches to improve their performance. Others have used knowledge acquisition to obtain optimal fuzzy rule set and genetic reinforcement learning (GRL) for designing consequent parts of fuzzy systems. We also propose a Q-value-based GRL for fuzzy controller (QGRF) where evolution is performed after each trial in contrast to GA where many trials are required to be performed before evolution.

[1]  Shingo Mabu,et al.  Adaptability analysis of genetic network programming with reinforcement learning in dynamically changing environments , 2012, Expert Syst. Appl..

[2]  Zhijng Liu,et al.  An Adaptive Ant Colony Optimization Algorithm Approach to Reinforcement Learning , 2008, 2008 International Symposium on Computational Intelligence and Design.

[3]  Chin-Teng Lin,et al.  GA-based fuzzy reinforcement learning for control of a magnetic bearing system , 2000, IEEE Trans. Syst. Man Cybern. Part B.

[4]  Hassan Bevrani,et al.  Load–frequency control : a GA-based multi-agent reinforcement learning , 2010 .

[5]  Chin-Teng Lin,et al.  Genetic Reinforcement Learning through Symbiotic Evolution for Fuzzy Controller Design , 2022 .

[6]  Chee Peng Lim,et al.  A new Reinforcement Learning-based Memetic Particle Swarm Optimizer , 2016, Appl. Soft Comput..

[7]  Hung-Yuan Chung,et al.  A self-learning fuzzy logic controller using genetic algorithms with reinforcements , 1997, IEEE Trans. Fuzzy Syst..

[8]  A. J. Yuste,et al.  Knowledge Acquisition in Fuzzy-Rule-Based Systems With Particle-Swarm Optimization , 2010, IEEE Transactions on Fuzzy Systems.

[9]  Cheng-Jian Lin,et al.  A novel genetic reinforcement learning for nonlinear fuzzy control problems , 2006, Neurocomputing.

[10]  Zhang Yi,et al.  Multiobjective Reinforcement Learning for Traffic Signal Control Using Vehicular Ad Hoc Network , 2010, EURASIP J. Adv. Signal Process..

[11]  Sheng-Fuu Lin,et al.  Reinforcement Hybrid Evolutionary Learning for TSK-Type Neuro-Fuzzy Controller Design , 2008 .

[12]  B. K. Panigrahi,et al.  Intelligent-based multi-robot path planning inspired by improved classical Q-learning and improved particle swarm optimization with perturbed velocity , 2016 .

[13]  Guangzhou Zeng,et al.  Study of genetic algorithm with reinforcement learning to solve the TSP , 2009, Expert Syst. Appl..

[14]  Madan Gopal,et al.  A REINFORCEMENT LEARNING ALGORITHM WITH EVOLVING FUZZY NEURAL NETWORKS , 2014 .

[15]  Anupam Shukla,et al.  Improved Approach to Area Exploration in an Unknown Environment by Mobile Robot using Genetic Algorithm, Real time Reinforcement Learning and Co-operation among the Controllers , 2014 .

[16]  Sheng-Fuu Lin,et al.  Reinforcement evolutionary learning using data mining algorithm with TSK-type fuzzy controllers , 2011, Appl. Soft Comput..

[17]  Chin-Teng Lin,et al.  Controlling chaos by GA-based reinforcement learning neural network , 1999, IEEE Trans. Neural Networks.

[18]  Chia-Feng Juang,et al.  Combination of online clustering and Q-value based GA for reinforcement fuzzy system design , 2005, IEEE Trans. Fuzzy Syst..

[19]  L. Lebensztajn,et al.  Non-Dominated Sorting Genetic Algorithm Based on Reinforcement Learning to Optimization of Broad-Band Reflector Antennas Satellite , 2012, IEEE Transactions on Magnetics.

[20]  David J. Williams,et al.  Nonconventional control of the flexible pole-cart balancing problem: Experimental results , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[21]  Chee Peng Lim,et al.  A hybrid model of fuzzy ARTMAP and genetic algorithm for data classification and rule extraction , 2016, Expert Syst. Appl..

[22]  Hak-Keung Lam,et al.  Design and stability analysis of fuzzy model-based nonlinear controller for nonlinear systems using genetic algorithm , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[23]  Changjiu Zhou,et al.  Robot learning with GA-based fuzzy reinforcement learning agents , 2002, Inf. Sci..

[24]  Hitoshi Iba,et al.  Adaptation technique for integrating genetic programming and reinforcement learning for real robots , 2005, IEEE Transactions on Evolutionary Computation.

[25]  Chia-Feng Juang,et al.  Ant Colony Optimization Incorporated With Fuzzy Q-Learning for Reinforcement Fuzzy Control , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[26]  Cheng-Hung Chen,et al.  Nonlinear system control using self-evolving neural fuzzy inference networks with reinforcement evolutionary learning , 2011, Appl. Soft Comput..

[27]  Chia-Feng Juang,et al.  Reinforcement Interval Type-2 Fuzzy Controller Design by Online Rule Generation and Q-Value-Aided Ant Colony Optimization , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Nikhil R. Pal,et al.  SOGARG: A self-organized genetic algorithm-based rule generation scheme for fuzzy controllers , 2003, IEEE Trans. Evol. Comput..