Joint Attention for Multi-Agent Coordination and Social Learning

Joint attention—the ability to purposefully coordinate attention with another agent, and mutually attend to the same thing—is a critical component of human social cognition. In this paper, we ask whether joint attention can be useful as a mechanism for improving multi-agent coordination and social learning. We first develop deep reinforcement learning (RL) agents with a recurrent visual attention architecture. We then train agents to minimize the difference between the attention weights that they apply to the environment at each timestep, and the attention of other agents. Our results show that this joint attention incentive improves agents’ ability to solve difficult coordination tasks, by reducing the exponential cost of exploring the joint multi-agent action space. Joint attention leads to higher performance than a competitive centralized critic baseline across multiple environments. Further, we show that joint attention enhances agents’ ability to learn from experts present in their environment, even when completing hard exploration tasks that do not require coordination. Taken together, these findings suggest that joint attention may be a useful inductive bias for multi-agent learning.

[1]  Tetsuo Ono,et al.  Physical relation and expression: joint attention for human-robot interaction , 2003, IEEE Trans. Ind. Electron..

[2]  H. Francis Song,et al.  Relational Forward Models for Multi-Agent Learning , 2018, ICLR.

[3]  H. Kozima,et al.  A Robot that Learns to Communicate with Human Caregivers , 2001 .

[4]  Guy Lever,et al.  Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward , 2018, AAMAS.

[5]  Tamer Basar,et al.  Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms , 2019, Handbook of Reinforcement Learning and Control.

[6]  Matthew W. Hoffman,et al.  A probabilistic model of gaze imitation and shared attention , 2006, Neural Networks.

[7]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[8]  Dawn Xiaodong Song,et al.  Assessing Generalization in Deep Reinforcement Learning , 2018, ArXiv.

[9]  Shimon Whiteson,et al.  Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? , 2020, ArXiv.

[10]  Ian R. Fasel,et al.  The emergence of shared attention: Using robots to test developmental theories , 2001 .

[11]  Shimon Whiteson,et al.  Counterfactual Multi-Agent Policy Gradients , 2017, AAAI.

[12]  Alexander Peysakhovich,et al.  Prosocial Learning Agents Solve Generalized Stag Hunts Better than Selfish Ones Extended Abstract , 2018 .

[13]  S. Levine,et al.  Learning Social Learning , 2020 .

[14]  Mykel J. Kochenderfer,et al.  Cooperative Multi-agent Control Using Deep Reinforcement Learning , 2017, AAMAS Workshops.

[15]  M. Tomasello.  Joint attention as social cognition. , 1995 .

[16]  Jürgen Schmidhuber,et al.  Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) , 2010, IEEE Transactions on Autonomous Mental Development.

[17]  Adina Magda Florea,et al.  Learning to Maximize Return in a Stag Hunt Collaborative Scenario through Deep Reinforcement Learning , 2017, 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC).

[18]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[19]  Jyh-Charn Liu,et al.  Translating Mathematical Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training , 2019, ArXiv.

[20]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[21]  Jongeun Choi,et al.  Distributed learning and cooperative control for multi-agent systems , 2009, Autom..

[22]  Riichiro Tadakuma,et al.  Towards shared attention through geometric reasoning for Human Robot Interaction , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[23]  Tiejun Huang,et al.  Graph Convolutional Reinforcement Learning , 2020, ICLR.

[24]  Craig Boutilier,et al.  The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems , 1998, AAAI/IAAI.

[25]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[26]  Alex Mott,et al.  Towards Interpretable Reinforcement Learning Using Attention Augmented Agents , 2019, NeurIPS.

[27]  Marlos C. Machado,et al.  Generalization and Regularization in DQN , 2018, ArXiv.

[28]  Minoru Asada,et al.  A constructive model for the development of joint attention , 2003, Connect. Sci..

[29]  Byoung-Tak Zhang,et al.  Multi-focus Attention Network for Efficient Deep Reinforcement Learning , 2017, AAAI Workshops.

[30]  Jun Ota,et al.  Multi-agent robot systems as distributed autonomous systems , 2006, Adv. Eng. Informatics.

[31]  P. Mundy,et al.  EEG correlates of the development of infant joint attention skills. , 2000, Developmental psychobiology.

[32]  Taehoon Kim,et al.  Quantifying Generalization in Reinforcement Learning , 2018, ICML.

[33]  Matthew E. Taylor,et al.  A survey and critique of multiagent deep reinforcement learning , 2019, Autonomous Agents and Multi-Agent Systems.

[34]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[35]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[36]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[37]  Nando de Freitas,et al.  Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning , 2018, ICML.

[38]  P. Mundy,et al.  A longitudinal study of joint attention and language development in autistic children , 1990, Journal of autism and developmental disorders.

[39]  Lars Schmidt-Thieme,et al.  Learning to Solve Vehicle Routing Problems with Time Windows through Joint Attention , 2020, ArXiv.

[40]  Shimon Whiteson,et al.  Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning , 2020, J. Mach. Learn. Res..

[41]  C. Moore,et al.  Joint attention : its origins and role in development , 1995 .

[42]  Andrea Lockerd Thomaz,et al.  An embodied computational model of social referencing , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[43]  Kunikazu Kobayashi,et al.  Cooperative Behavior Acquisition in Multi-agent Reinforcement Learning System Using Attention Degree , 2012, ICONIP.

[44]  Shimon Whiteson,et al.  QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning , 2018, ICML.

[45]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[46]  F. Kaplan,et al.  The challenges of joint attention , 2006 .

[47]  Quoc V. Le,et al.  Attention Augmented Convolutional Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Andrea Lockerd Thomaz,et al.  Joint Attention in Human-Robot Interaction , 2010, AAAI Fall Symposium: Dialog with Robots.

[49]  J. Triesch,et al.  A computational model of the emergence of gaze following , 2004 .

[50]  Rob Fergus,et al.  Modeling Others using Oneself in Multi-Agent Reinforcement Learning , 2018, ICML.

[51]  Nuttapong Chentanez,et al.  Intrinsically Motivated Reinforcement Learning , 2004, NIPS.

[52]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[53]  B. Scassellati.  Imitation and mechanisms of joint attention: a developmental structure for building social skills on a humanoid robot , 1999 .

[54]  Daniel Kudenko,et al.  MAGNet: Multi-agent Graph Network for Deep Multi-agent Reinforcement Learning , 2019, 2019 XVI International Symposium "Problems of Redundancy in Information and Control Systems" (REDUNDANCY).

[55]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[56]  Yujing Hu,et al.  Multi-Agent Game Abstraction via Graph Attention Neural Network , 2019, AAAI.