Hybrid Game Strategy in Fuzzy Markov-Game-Based Control

This paper proposes a novel control approach that incorporates a hybrid game strategy in Markov-game-based fuzzy control. Specifically, we aim at designing a ldquosafe and universally consistentrdquo controller that exhibits an ability to maintain performance against large disturbance and environment variations. The proposed hybrid control is a convex combination (based on experiential information) of ldquoa variation of cautious fictitious playrdquo approach and the ldquominimaxrdquo control approach implemented on a fuzzy Markov game platform. We show analytical convergence of Markov-game-based control in the presence of bounded external disturbances, and extend the analysis to show convergence of the proposed Markov-game-based hybrid control approach. Controller simulation and comparison against baseline Markov game fuzzy control and fuzzy Q -learning control on a highly nonlinear two-link robot brings out the superiority of the approach in handling severe environment and disturbance variations over different desired trajectories. This paper illustrates the possibility of obtaining ldquouniversal consistency,rdquo i.e., reasonable performance against severe environment and disturbance variations, by hybridizing ldquocautious fictitious playrdquo with ldquominimaxrdquo approaches in Markov-game-based control.

[1]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 2005, IEEE Transactions on Neural Networks.

[2]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[3]  Frank L. Lewis,et al.  Direct-reinforcement-adaptive-learning neural network control for nonlinear systems , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[4]  Rajneesh Sharma,et al.  A robust Markov game controller for nonlinear systems , 2007, Appl. Soft Comput..

[5]  G. Owen,et al.  Game Theory (2nd Ed.). , 1983 .

[6]  Rajneesh Sharma,et al.  A Safe and Consistent Game-theoretic Controller for Nonlinear Systems , 2005, IICAI.

[7]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[8]  Chuan-Kai Lin,et al.  A reinforcement learning adaptive fuzzy controller for robots , 2003, Fuzzy Sets Syst..

[9]  Warren B. Powell,et al.  Handbook of Learning and Approximate Dynamic Programming , 2006, IEEE Transactions on Automatic Control.

[10]  D. Fudenberg,et al.  Consistency and Cautious Fictitious Play , 1995 .

[11]  Rajneesh Sharma,et al.  A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators , 2008, IEEE Transactions on Fuzzy Systems.

[12]  Frank L. Lewis,et al.  Neural Network Control Of Robot Manipulators And Non-Linear Systems , 1998 .

[13]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[14]  Jun Morimoto,et al.  Robust Reinforcement Learning , 2005, Neural Computation.

[15]  Michael L. Littman,et al.  Algorithms for Sequential Decision Making , 1996 .

[16]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[17]  Lionel Jouffe,et al.  Fuzzy inference system learning by reinforcement methods , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[18]  Richard S. Sutton,et al.  Reinforcement Learning is Direct Adaptive Optimal Control , 1992, 1991 American Control Conference.

[19]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[20]  P. Glorennec,et al.  Fuzzy Q-learning , 1997, Proceedings of 6th International Fuzzy Systems Conference.