This paper introduces an information-theoretic constraint on learned policy complexity in the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm. Previous research with a related approach in continuous control experiments suggests that this method favors learning policies that are more robust to changing environment dynamics. The multi-agent game setting naturally requires this type of robustness, as other agents' policies change throughout learning, introducing a nonstationary environment. For this reason, recent methods in continual learning are compared to our approach, termed Capacity-Limited MADDPG. Results from experimentation in multi-agent cooperative and competitive tasks demonstrate that the capacity-limited approach is a good candidate for improving learning performance in these environments.
[1]
Sergey Levine,et al.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
,
2018,
ICML.
[2]
Murray Shanahan,et al.
Policy Consolidation for Continual Reinforcement Learning
,
2019,
ICML.
[3]
Stefan Wermter,et al.
Continual Lifelong Learning with Neural Networks: A Review
,
2019,
Neural Networks.
[4]
Yi Wu,et al.
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
,
2017,
NIPS.
[5]
Gerald Tesauro,et al.
Deep RL With Information Constrained Policies: Generalization in Continuous Control
,
2020,
ArXiv.
[6]
Craig Boutilier,et al.
The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems
,
1998,
AAAI/IAAI.
[7]
Jordi Grau-Moya,et al.
Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
,
2019,
CoRL.