Kernel Recursive Least Squares Function Approximation in Game Theory Based Control

A game theoretic aspect in reinforcement learning based controller design with kernel recursive least squares algorithm for value function approximation is proposed in this paper. A kernel recursive least-squares-support vector machine is used to realize a mapping from state, controller's action and disturber's action to Q-value function. Online sparsification framework permits the addition of training sample into the Q-function approximation only if it is approximately linearly independent of the preceding training samples. Markov game setup is shown to be one of the important platforms for addressing robustness of direct adaptive optimal control of nonlinear systems. A game against nature strategy shows the strength of state importance in terms of accelerated learning, and better relative stability of the system. Simulation results on two-link robot manipulator show that the proposed method has high learning efficiency—better accuracy measured in terms of mean square error; and lesser computation time, compared to the least-squares support vector machine.

[1]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[2]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[3]  R. Brereton,et al.  Support vector machines for classification and regression. , 2010, The Analyst.

[4]  Richard S. Sutton,et al.  Temporal credit assignment in reinforcement learning , 1984 .

[5]  Anthony Green,et al.  Dynamics and Trajectory Tracking Control of a Two-Link Robot Manipulator , 2004 .

[6]  Rajneesh Sharma,et al.  A robust Markov game controller for nonlinear systems , 2007, Appl. Soft Comput..

[7]  Rajneesh Sharma,et al.  A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators , 2008, IEEE Transactions on Fuzzy Systems.

[8]  Michael L. Littman,et al.  Value-function reinforcement learning in Markov games , 2001, Cognitive Systems Research.

[9]  M. Gopal,et al.  A fuzzy decision tree-based robust Markov game controller for robot manipulators , 2010, Int. J. Autom. Control..

[10]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[11]  S. Gunn.  Support Vector Machines for Classification and Regression , 1998 .

[12]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[13]  Michael L. Littman,et al.  Algorithms for Sequential Decision Making , 1996 .

[14]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[15]  Xuesong Wang,et al.  Value Approximation with Least Squares Support Vector Machine in Reinforcement Learning System , 2007 .