Emergent Multi-Agent Communication in the Deep Learning Era

The ability to cooperate through language is a defining feature of humans. As the perceptual, motory and planning capabilities of deep artificial networks increase, researchers are studying whether they also can develop a shared language to interact. From a scientific perspective, understanding the conditions under which language evolves in communities of deep agents and its emergent features can shed light on human language evolution. From an applied perspective, endowing deep networks with the ability to solve problems interactively by communicating with each other and with us should make them more flexible and useful in everyday life. This article surveys representative recent language emergence studies from both of these two angles.

[1]  Anna Maria Di Sciullo.  On Aspects of the Theory of Syntax , 2021, Inference: International Review of Science.

[2]  Angeliki Lazaridou,et al.  Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning , 2020, ACL.

[3]  Eugene Kharitonov,et al.  Compositionality and Generalization In Emergent Languages , 2020, ACL.

[4]  Aaron C. Courville,et al.  Countering Language Drift with Seeded Iterated Learning , 2020, ICML.

[5]  Joelle Pineau,et al.  On the interaction between supervision and self-play in emergent communication , 2020, ICLR.

[6]  Andrew M. Dai,et al.  Capacity, Bandwidth, and Compositionality in Emergent Language Learning , 2019, AAMAS.

[7]  Jason J. Corso,et al.  Unified Vision-Language Pre-Training for Image Captioning and VQA , 2019, AAAI.

[8]  Marco Baroni,et al.  Entropy Minimization In Emergent Languages , 2019, ICML.

[9]  H. Francis Song,et al.  The Hanabi Challenge: A New Frontier for AI Research , 2019, Artif. Intell..

[10]  Territoire Urbain,et al.  Convention , 1955, Hidden Nature.

[11]  Doina Precup,et al.  Shaping representations through communication: community size effect in artificial learning systems , 2019, ArXiv.

[12]  Tom Eccles,et al.  Biases for Emergent Communication in Multi-agent Reinforcement Learning , 2019, NeurIPS.

[13]  Anca D. Dragan,et al.  On the Utility of Learning about Humans for Human-AI Coordination , 2019, NeurIPS.

[14]  Kyunghyun Cho,et al.  Countering Language Drift via Visual Grounding , 2019, EMNLP.

[15]  Shiri Lev-Ari,et al.  Larger communities create more systematic languages , 2019, Proceedings of the Royal Society B.

[16]  Eugene Kharitonov,et al.  EGG: a toolkit for research on Emergence of lanGuage in Games , 2019, EMNLP.

[17]  Michael Bowling,et al.  Ease-of-Teaching and Language Structure from Emergent Communication , 2019, NeurIPS.

[18]  Jianfeng Gao,et al.  Neural Approaches to Conversational AI: Question Answering, Task-oriented Dialogues and Social Chatbots , 2019 .

[19]  Eugene Kharitonov,et al.  Anti-efficient encoding in emergent communication , 2019, NeurIPS.

[20]  Marco Baroni,et al.  Miss Tools and Mr Fruit: Emergent Communication in Agents Learning about Object Affordances , 2019, ACL.

[21]  David C. Paris.  Capacity , 2019, Change: The Magazine of Higher Learning.

[22]  E. Gibson,et al.  How Efficiency Shapes Human Language , 2019, Trends in Cognitive Sciences.

[23]  Joelle Pineau,et al.  On the Pitfalls of Measuring Emergent Communication , 2019, AAMAS.

[24]  Jacob Andreas,et al.  Measuring Compositionality in Representation Learning , 2019, ICLR.

[25]  Taeyoung Lee,et al.  Learning to Schedule Communication in Multi-agent Reinforcement Learning , 2019, ICLR.

[26]  Laura Graesser,et al.  Emergent Linguistic Phenomena in Multi-Agent Communication Games , 2019, EMNLP.

[27]  Shiri Lev-Ari,et al.  Compositional structure can emerge without generational transmission , 2019, Cognition.

[28]  H. Francis Song,et al.  Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning , 2018, ICML.

[29]  Stefan Bauer,et al.  Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness , 2018, ICML.

[30]  Nando de Freitas,et al.  Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning , 2018, ICML.

[31]  Amanpreet Singh,et al.  Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks , 2018, ICLR.

[32]  Joelle Pineau,et al.  TarMAC: Targeted Multi-Agent Communication , 2018, ICML.

[33]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[34]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[35]  Jonathan Berant,et al.  Emergence of Communication in an Interactive World with Consistent Speakers , 2018, ArXiv.

[36]  Marco Baroni,et al.  How agents see things: On visual representations in an emergent language game , 2018, EMNLP.

[37]  Myle Ott,et al.  Understanding Back-Translation at Scale , 2018, EMNLP.

[38]  K. Zuberbühler,et al.  Compositionality in animals and humans , 2018, PLoS biology.

[39]  Nando de Freitas,et al.  Compositional Obverter Communication Learning From Raw Visual Input , 2018, ICLR.

[40]  Stephen Clark,et al.  Emergent Communication through Negotiation , 2018, ICLR.

[41]  Stephen Clark,et al.  Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input , 2018, ICLR.

[42]  Lei Zhang,et al.  Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Kyunghyun Cho,et al.  Emergent Communication in a Multi-Modal, Multi-Step Referential Game , 2017, ICLR.

[44]  Pieter Abbeel,et al.  Emergence of Grounded Compositional Language in Multi-Agent Populations , 2017, AAAI.

[45]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[46]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[47]  José M. F. Moura,et al.  Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog , 2017, EMNLP.

[48]  Iyad Rahwan,et al.  Cooperating with machines , 2017, Nature Communications.

[49]  Ivan Titov,et al.  Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols , 2017, NIPS.

[50]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[51]  Alexander Peysakhovich,et al.  Multi-Agent Cooperation and the Emergence of (Natural) Language , 2016, ICLR.

[52]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[53]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[54]  Jonathan Ginzburg,et al.  Grammar Is a System That Characterizes Talk in Interaction , 2016, Front. Psychol..

[55]  Tamas David-Barrett,et al.  Language as a coordination tool evolves slowly , 2016, Royal Society Open Science.

[56]  Joelle Pineau,et al.  Generative Deep Neural Networks for Dialogue: A Short Review , 2016, ArXiv.

[57]  Emil Gustavsson,et al.  Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence , 2016, ArXiv.

[58]  Michael C. Frank,et al.  Review Pragmatic Language Interpretation as Probabilistic Inference , 2022 .

[59]  Rob Fergus,et al.  Learning Multiagent Communication with Backpropagation , 2016, NIPS.

[60]  Shimon Whiteson,et al.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning , 2016, NIPS.

[61]  Gary Lupyan,et al.  How Language Programs the Mind , 2016, Top. Cogn. Sci..

[62]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[63]  Gemma Boleda,et al.  Distributional Semantics in Use , 2015, LSDSem@EMNLP.

[64]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[65]  David DeVault,et al.  Toward Natural Turn-Taking in a Virtual Human Negotiation Agent , 2015, AAAI Spring Symposia.

[66]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[67]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[68]  S. Kirby,et al.  Iterated learning and the evolution of language , 2014, Current Opinion in Neurobiology.

[69]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[70]  D. Bickerton.  More Than Nature Needs: Language, Mind, and Evolution , 2014 .

[71]  付伶俐.  打磨Using Language,倡导新理念 , 2014 .

[72]  David Lusseau,et al.  Compression as a Universal Principle of Animal Behavior , 2013, Cogn. Sci..

[73]  M. Engelmann.  The Philosophical Investigations , 2013 .

[74]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[75]  Armin W. Schulz.  Signals: evolution, learning, and information , 2012 .

[76]  Luc Steels,et al.  Experiments in cultural language evolution , 2012 .

[77]  Mikael Parkvall,et al.  Creoles are typologically distinct from non-creoles , 2011 .

[78]  Michael A. Goodrich,et al.  Learning to compete, coordinate, and cooperate in repeated games using reinforcement learning , 2011, Machine Learning.

[79]  Richard L. Lewis,et al.  A new approach to exploring language emergence as boundedly optimal control in the face of environmental and cognitive constraints , 2010 .

[80]  Per Linell.  Rethinking Language, Mind and World Dialogically : Interactional and contextual theories of human sense-making , 2009 .

[81]  Csr Young,et al.  How to Do Things With Words , 2009 .

[82]  A. Kamiya,et al.  Learning of communication codes in multi-agent reinforcement learning problem , 2008, 2008 IEEE Conference on Soft Computing in Industrial Applications.

[83]  N. Masataka.  The Origins of Language , 2008 .

[84]  M. Tomasello.  Origins of human communication , 2008 .

[85]  M. Tomasello,et al.  Humans Have Evolved Specialized Skills of Social Cognition: The Cultural Intelligence Hypothesis , 2007, Science.

[86]  Gabriel Altmann,et al.  Word Length and Word Frequency , 2007 .

[87]  Simon Kirby,et al.  Understanding Linguistic Evolution by Visualizing the Emergence of Topographic Mappings , 2006, Artificial Life.

[88]  Sean Luke,et al.  Cooperative Multi-Agent Learning: The State of the Art , 2005, Autonomous Agents and Multi-Agent Systems.

[89]  J. Morgan,et al.  Cheap Talk , 2005 .

[90]  Siobhan Chapman.  Logic and Conversation , 2005 .

[91]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[92]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[93]  Sonia Martínez,et al.  Coverage control for mobile sensing networks , 2002, IEEE Transactions on Robotics and Automation.

[94]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[95]  L. Steels.  Evolving grounded communication for robots , 2003, Trends in Cognitive Sciences.

[96]  James A. Reggia,et al.  Progress in the Simulation of Emergent Communication and Language , 2003, Adapt. Behav..

[97]  S. Kirby,et al.  The emergence of linguistic structure: an overview of the iterated learning model , 2002 .

[98]  Carol Myers-Scotton,et al.  Contact Linguistics: Bilingual encounters and grammatical outcomes , 2013 .

[99]  Angelo Cangelosi,et al.  Simulating the Evolution of Language , 2002, Springer London.

[100]  Ivan A. Sag,et al.  Syntactic Theory: A Formal Introduction , 1999, Computational Linguistics.

[101]  M. E. Medina-Callarotti.  Origins of Language , 2000 .

[102]  M. Marchesi,et al.  Scaling and criticality in a stochastic multi-agent model of a financial market , 1999, Nature.

[103]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[104]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[105]  R. Morse.  The Dance Language and Orientation of Bees , 1994 .

[106]  S. Pinker.  The Language Instinct , 1994 .

[107]  Ming Tan,et al.  Multi-Agent Reinforcement Learning: Independent versus Cooperative Agents , 1997, ICML.

[108]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[109]  Murray L Weidenbaum,et al.  Learning to compete , 1986 .

[110]  W. Güth,et al.  An experimental analysis of ultimatum bargaining , 1982 .

[111]  J. Sobel,et al.  STRATEGIC INFORMATION TRANSMISSION , 1982 .

[112]  J. Allwood.  Linguistic communication as action and cooperation : a study in pragmatics , 1976 .

[113]  A. Koller,et al.  Speech Acts: An Essay in the Philosophy of Language , 1969 .

[114]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[115]  R. Paget.  The Origin of Speech , 1927, Nature.