An approach to tune fuzzy controllers based on reinforcement learning for autonomous vehicle control

In this paper, we suggest a new approach for tuning parameters of fuzzy controllers based on reinforcement learning. The architecture of the proposed approach is comprised of a Q estimator network (QEN) and a Takagi-Sugeno-type fuzzy inference system (TSK-FIS). Unlike other fuzzy Q-learning approaches that select an optimal action based on finite discrete actions, the proposed controller obtains the control output directly from TSK-FIS. With the proposed architecture, the learning algorithms for all the parameters of the QEN and the FIS are developed based on the temporal-difference (TD) methods as well as the gradient-descent algorithm. The performance of the proposed design technique is illustrated by simulation studies of a vehicle longitudinal-control system.

[1]  Hung-Yuan Chung,et al.  A self-learning fuzzy logic controller using genetic algorithms with reinforcements , 1997, IEEE Trans. Fuzzy Syst..

[2]  Chin-Teng Lin,et al.  A reinforcement neuro-fuzzy combiner for multiobjective control , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[3]  Vijaykumar Gullapalli,et al.  A stochastic reinforcement learning algorithm for learning real-valued functions , 1990, Neural Networks.

[4]  Martial Hebert,et al.  Intelligent Unmanned Ground Vehicles , 1997 .

[5]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[6]  Jennie Si,et al.  The best approximation to C2 functions and its error bounds using regular-center Gaussian networks , 1994, IEEE Trans. Neural Networks.

[7]  Swaroop Darbha,et al.  Direct adaptive longitudinal control of vehicle platoons , 2001, IEEE Trans. Veh. Technol..

[8]  Chin-Teng Lin,et al.  Reinforcement structure/parameter learning for neural-network-based fuzzy logic control systems , 1993, [Proceedings 1993] Second IEEE International Conference on Fuzzy Systems.

[9]  Jae-Hyun Kim,et al.  Fuzzy-Q learning for autonomous robot systems , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[10]  Jennie Si,et al.  Online learning control by association and reinforcement. , 2001, IEEE transactions on neural networks.

[11]  Jeich Mar,et al.  An ANFIS controller for the car-following collision prevention system , 2001, IEEE Trans. Veh. Technol..

[12]  Nick McKeown,et al.  Automated vehicle control developments in the PATH program , 1991 .

[13]  Nasser Kehtarnavaz,et al.  A transportable neural-network approach to autonomous vehicle following , 1998 .

[14]  K. Kurami,et al.  The development of autonomously controlled vehicle, PVS , 1991, Vehicle Navigation and Information Systems Conference, 1991.

[15]  José Eugenio Naranjo,et al.  Adaptive fuzzy control for inter-vehicle gap keeping , 2003, IEEE Trans. Intell. Transp. Syst..

[16]  Chin-Teng Lin,et al.  Reinforcement learning for an ART-based fuzzy adaptive learning control network , 1996, IEEE Trans. Neural Networks.

[17]  Z. Deng,et al.  Competitive Takagi-Sugeno fuzzy reinforcement learning , 2001, Proceedings of the 2001 IEEE International Conference on Control Applications (CCA'01) (Cat. No.01CH37204).

[18]  J. Overholt,et al.  Intelligent unmanned ground vehicles , 2004 .

[19]  Gerald Tesauro,et al.  TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play , 1994, Neural Computation.

[20]  Andrew W. Moore,et al.  Generalization in Reinforcement Learning: Safely Approximating the Value Function , 1994, NIPS.

[21]  Andrew W. Moore,et al.  Gradient Descent for General Reinforcement Learning , 1998, NIPS.

[22]  Leslie Pack Kaelbling,et al.  Practical Reinforcement Learning in Continuous Spaces , 2000, ICML.

[23]  Il Hong Suh,et al.  Region-based Q-learning for intelligent robot systems , 1997, Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation'.

[24]  Bart Kosko,et al.  Fuzzy throttle and brake control for platoons of smart cars , 1996, Fuzzy Sets Syst..

[25]  Se-Young Oh,et al.  A new reinforcement learning vehicle control architecture for vision-based road following , 2000, IEEE Trans. Veh. Technol..

[26]  L. C. Baird,et al.  Reinforcement learning in continuous time: advantage updating , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[27]  Lionel Jouffe,et al.  Fuzzy inference system learning by reinforcement methods , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[28]  Yasuharu Koike,et al.  Multiple state estimation reinforcement learning for driving model: driver model of automobile , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[29]  Kwang-Ick Kim,et al.  Connectionist-nonconnectionist fusion architecture for high speed road following , 1996, Neural Parallel Sci. Comput..

[30]  Eduardo Zalama Casanova,et al.  Adaptive behavior navigation of a mobile robot , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[31]  Li-Xin Wang.  Stable adaptive fuzzy control of nonlinear systems , 1993, IEEE Trans. Fuzzy Syst..

[32]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[33]  R.J. Williams,et al.  Reinforcement learning is direct adaptive optimal control , 1991, IEEE Control Systems.

[34]  Wei Ren,et al.  Use of neural fuzzy networks with mixed genetic/gradient algorithm in automated vehicle control , 1999, IEEE Trans. Ind. Electron..

[35]  Martial Hebert,et al.  Intelligent Unmanned Ground Vehicles: Autonomous Navigation Research at Carnegie Mellon , 1997 .