Emergent Communication through Negotiation

Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve specific problems. In this paper, we study the emergence of communication in the negotiation environment, a semi-cooperative model of agent interaction. We introduce two communication protocols -- one grounded in the semantics of the game, and one which is \textit{a priori} ungrounded and is a form of cheap talk. We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly, but are unable to effectively use the ungrounded channel. However, prosocial agents do learn to use cheap talk to find an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge. We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identifiability can aid negotiation.

[1]  W. Güth,et al.  An experimental analysis of ultimatum bargaining , 1982 .

[2]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[3]  J. Nash.  NON-COOPERATIVE GAMES , 1951, Classics in Game Theory.

[4]  J. Nash.  THE BARGAINING PROBLEM , 1950, Classics in Game Theory.

[5]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[6]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[7]  Iyad Rahwan,et al.  Cooperating with machines , 2017, Nature Communications.

[8]  Alexander Peysakhovich,et al.  Multi-Agent Cooperation and the Emergence of (Natural) Language , 2016, ICLR.

[9]  Raymond J. Dolan,et al.  Game Theory of Mind , 2008, PLoS Comput. Biol..

[10]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[11]  T. Schelling,et al.  The Strategy of Conflict. , 1961 .

[12]  Shimon Whiteson,et al.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning , 2016, NIPS.

[13]  Hans Peters,et al.  Game Theory: A Multi-Leveled Approach , 2008 .

[14]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[15]  J. Neumann,et al.  Theory of Games and Economic Behavior. , 1945 .

[16]  A J Robson,et al.  Efficiency in evolutionary games: Darwin, Nash and the secret handshake. , 1990, Journal of theoretical biology.

[17]  Shimon Whiteson,et al.  Learning with Opponent-Learning Awareness , 2017, AAMAS.

[18]  Claudia V. Goldman,et al.  Learning to communicate in a decentralized environment , 2007, Autonomous Agents and Multi-Agent Systems.

[19]  Brian Skyrms,et al.  Signals, Evolution and the Explanatory Power of Transient Information* , 2002, Philosophy of Science.

[20]  R. J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[21]  Yann Dauphin,et al.  Deal or No Deal? End-to-End Learning of Negotiation Dialogues , 2017, EMNLP.

[22]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[23]  David Silver,et al.  A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning , 2017, NIPS.

[24]  David Lewis.  Convention: A Philosophical Study , 1986 .

[25]  David DeVault,et al.  Toward Natural Turn-Taking in a Virtual Human Negotiation Agent , 2015, AAAI Spring Symposia.

[26]  A. Rubinstein,et al.  The Nash bargaining solution in economic modelling , 1985 .

[27]  A. Rubinstein.  Perfect Equilibrium in a Bargaining Model , 1982 .

[28]  Alexander Peysakhovich,et al.  Prosocial Learning Agents Solve Generalized Stag Hunts Better than Selfish Ones Extended Abstract , 2018 .

[29]  Jakub W. Pachocki,et al.  Emergent Complexity via Multi-Agent Competition , 2017, ICLR.

[30]  M A Nowak,et al.  The evolution of language. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[31]  J. Morgan,et al.  Cheap Talk , 2005 .

[32]  Kyunghyun Cho,et al.  Emergent Language in a Multi-Modal, Multi-Step Referential Game , 2017, ArXiv.

[33]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[34]  Gerhard Weiss,et al.  Multiagent Learning: Basics, Challenges, and Prospects , 2012, AI Mag..

[35]  Ivan Titov,et al.  Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols , 2017, NIPS.

[36]  A. Copeland.  Review: John von Neumann and Oskar Morgenstern, Theory of games and economic behavior , 1945 .

[37]  Matthew Saffell,et al.  Learning to trade via direct reinforcement , 2001, IEEE Trans. Neural Networks.

[38]  Robert H. Crites,et al.  Multiagent reinforcement learning in the Iterated Prisoner's Dilemma. , 1996, Bio Systems.

[39]  J. Sobel,et al.  STRATEGIC INFORMATION TRANSMISSION , 1982 .

[40]  Rob Fergus,et al.  Learning Multiagent Communication with Backpropagation , 2016, NIPS.

[41]  Michael L. Littman,et al.  Classes of Multiagent Q-learning Dynamics with epsilon-greedy Exploration , 2010, ICML.

[42]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[43]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[44]  James A. Reggia,et al.  Progress in the Simulation of Emergent Communication and Language , 2003, Adapt. Behav..