A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators

This paper develops an adaptive fuzzy controller for robot manipulators using a Markov game formulation. The Markov game framework offers a promising platform for robust control of robot manipulators in the presence of bounded external disturbances and unknown parameter variations. We propose fuzzy Markov games as an adaptation of fuzzy Q-learning (FQL) to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online the conclusion part of a fuzzy Markov game controller. The proposed Markov game-adaptive fuzzy controller uses a simple fuzzy inference system (FIS), is computationally efficient, generates a swift control, and requires no exact dynamics of the robot system. To illustrate the superiority of Markov game-adaptive fuzzy control, we compare the performance of the controller against a) the Markov game-based robust neural controller, b) the reinforcement learning (RL)-adaptive fuzzy controller, c) the FQL controller, d) the Hinfin theory-based robust neural game controller, and e) a standard RL-based robust neural controller, on two highly nonlinear robot arm control problems of i) a standard two-link rigid robot arm and ii) a 2-DOF SCARA robot manipulator. The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories. The results also demonstrate the viability of FISs for accelerating learning in Markov games and extending Markov game-based control to continuous state-action space problems.

[1]  Richard S. Sutton,et al.  Reinforcement Learning is Direct Adaptive Optimal Control , 1992, 1991 American Control Conference.

[2]  P. Glorennec,et al.  Fuzzy Q-learning , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[3]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[4]  Chuan-Kai Lin,et al.  A reinforcement learning adaptive fuzzy controller for robots , 2003, Fuzzy Sets Syst..

[5]  Jun Morimoto,et al.  Robust Reinforcement Learning , 2005, Neural Computation.

[6]  L X Wang,et al.  Fuzzy basis functions, universal approximation, and orthogonal least-squares learning , 1992, IEEE Trans. Neural Networks.

[7]  Rajneesh Sharma,et al.  A robust Markov game controller for nonlinear systems , 2007, Appl. Soft Comput..

[8]  Michael L. Littman,et al.  Algorithms for Sequential Decision Making , 1996 .

[9]  Lionel Jouffe,et al.  Fuzzy inference system learning by reinforcement methods , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[11]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[12]  Frank L. Lewis,et al.  Direct-reinforcement-adaptive-learning neural network control for nonlinear systems , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[13]  Maarouf Saad,et al.  Adaptive versus neural adaptive-control : application to robotics , 1994 .

[14]  Matthias Heger,et al.  Consideration of Risk in Reinforcement Learning , 1994, ICML.

[15]  Michail G. Lagoudakis,et al.  Learning in Zero-Sum Team Markov Games Using Factored Value Functions , 2002, NIPS.

[16]  Michael L. Littman,et al.  Value-function reinforcement learning in Markov games , 2001, Cognitive Systems Research.

[17]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[18]  Michail G. Lagoudakis,et al.  Value Function Approximation in Zero-Sum Markov Games , 2002, UAI.