A study on multi-agent reinforcement learning problem based on hierarchical modular fuzzy model

Reinforcement learning is a promising approach to realize intelligent agent such as autonomous mobile robots. In order to apply the reinforcement learning to actual sized problem, the “curse of dimensionality” problem in partition of sensory states should be avoided maintaining computational efficiency. The paper describes a hierarchical modular reinforcement learning that Profit Sharing learning algorithm is combined with Q-Learning reinforcement learning algorithm hierarchically in multi-agent pursuit environment. As the model structure for such huge problem, I propose a modular fuzzy model extending SIRMs architecture. Through numerical experiments, I found that the proposed method has good convergence property of learning compared with the conventional algorithms.

[1]  P. Glorennec,et al.  Fuzzy Q-learning , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[2]  John J. Grefenstette,et al.  Credit assignment in rule discovery systems based on genetic algorithms , 1988, Machine Learning.

[3]  Minoru Asada,et al.  Behavior acquisition by multi-layered reinforcement learning , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[4]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[5]  Hironori Hirata,et al.  An adjustment method of the number of states on Q‐learning segmenting state space adaptively , 2007 .

[6]  Akira Ito,et al.  Speeding up Multi-Agent Reinforcement Learning by Coarse-Graining of Perception ——Hunter Game as an Example—— , 2001 .

[7]  Richard S. Sutton,et al.  Reinforcement Learning , 1992, Handbook of Machine Learning.

[8]  Kenji Fukumoto,et al.  Multi-agent Reinforcement Learning: A Modular Approach , 1996 .

[9]  M. Benda,et al.  On Optimal Cooperation of Knowledge Sources , 1985 .

[10]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[12]  Mohamed S. Kamel,et al.  Modular neural networks: a survey. , 1999, International journal of neural systems.

[13]  Jun Morimoto,et al.  Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning , 2000, Robotics Auton. Syst..

[14]  N. Yubazaki.  SIRMs dynamically connected fuzzy inference model and its applications , 1997 .

[15]  Hiroshi Matsuo,et al.  Multiagent reinforcement learning with the partly high-dimensional state space , 2005 .

[16]  Hiroaki Ishii,et al.  On the Generalization of Single Input Rule Modules Connected Type Fuzzy Reasoning Method , 2008, IEEE Transactions on Fuzzy Systems.

[17]  Toshihiko Watanabe,et al.  Learning of Agent Behavior Based on Hierarchical Modular Reinforcement Learning , 2006 .

[18]  Toshihiko Watanabe,et al.  Hierarchical reinforcement learning using a modular fuzzy model for multi-agent problem , 2007, 2007 IEEE International Conference on Systems, Man and Cybernetics.

[19]  Hiroshi Matsuo,et al.  Multiagent reinforcement learning with the partly high-dimensional state space , 2006, Systems and Computers in Japan.

[20]  Shinzo Kitamura,et al.  Q-Learning with adaptive state segmentation (QLASS) , 1997, Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation'.