论文信息 - A reinforcement learning model using macro-actions in multi-task grid-world problems

A reinforcement learning model using macro-actions in multi-task grid-world problems

A macro-action is a typical series of useful actions that brings high expected rewards to an agent. Murata et al. have proposed an Actor-Critic model which can generate macro-actions automatically based on the information on state values and visiting frequency of states. However, their model has not assumed that generated macro-actions are utilized for leaning different tasks. In this paper, we extend the Murata's model such that generated macro-actions can help an agent learn an optimal policy quickly in multi-task Grid-World (MTGW) maze problems. The proposed model is applied to two MTGW problems, each of which consists of six different maze tasks. From the experimental results, it is concluded that the proposed model could speed up learning if macro-actions are generated in the so-called correlated regions.

Seiichi Ozawa | Hiroshi Onda

[1] Sebastian Thrun,et al. Learning to Learn , 1998, Springer US.

[2] Shigeo Abe,et al. A reinforcement learning algorithm for neural networks with incremental learning ability , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[3] Glenn A. Iba,et al. A heuristic approach to the discovery of macro-operators , 2004, Machine Learning.

[4] Seiichi Ozawa,et al. A Memory-Based Reinforcement Learning Model Utilizing Macro-Actions , 2005 .

[5] Andrew G. Barto,et al. Building Portable Options: Skill Transfer in Reinforcement Learning , 2007, IJCAI.

[6] Richard S. Sutton,et al. Roles of Macro-Actions in Accelerating Reinforcement Learning , 1998 .

[7] Andrew G. Barto,et al. Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density , 2001, ICML.