Incorporating Pragmatic Reasoning Communication into Emergent Language

Emergentism and pragmatics are two research fields that study the dynamics of linguistic communication along substantially different timescales and intelligence levels. From the perspective of multi-agent reinforcement learning, they correspond to stochastic games with reinforcement training and stage games with opponent awareness. Given that their combination has been explored in linguistics, we propose computational models that combine short-term mutual reasoning-based pragmatics with long-term language emergentism. We explore this for agent communication referential games as well as in Starcraft II, assessing the relative merits of different kinds of mutual reasoning pragmatics models both empirically and theoretically. Our results shed light on their importance for making inroads towards getting more natural, accurate, robust, fine-grained, and succinct utterances.

[1]  G. Zipf,et al.  The Psycho-Biology of Language , 1936 .

[2]  G. Āllport.  The Psycho-Biology of Language. , 1936 .

[3]  R. Kirk.  CONVENTION: A PHILOSOPHICAL STUDY , 1970 .

[4]  David Lewis.  Convention: A Philosophical Study , 1986 .

[5]  H. V. VAN TONGEREN.  [The use of language]. , 1986, Tijdschrift voor diergeneeskunde.

[6]  G. Zipf.  The Psycho-Biology Of Language: AN INTRODUCTION TO DYNAMIC PHILOLOGY , 1999 .

[7]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[8]  Geoffrey Finch.  Key Concepts in Language and Linguistics , 2005 .

[9]  Siobhan Chapman.  Logic and Conversation , 2005 .

[10]  Ariel Rubinstein,et al.  A Game Theoretic Approach to the Pragmatics of Debate: An Expository Note , 2006 .

[11]  Michael Franke,et al.  Signal to act : game theory in pragmatics , 2009 .

[12]  Robin Lee Clark,et al.  Meaningful Games: Exploring Language with Game Theory , 2011 .

[13]  M. Dufwenberg.  Game theory. , 2011, Wiley interdisciplinary reviews. Cognitive science.

[14]  Christiane Fellbaum,et al.  Empirical Comparisons of MASC Word Sense Annotations , 2012, LREC.

[15]  A. Goldman.  To Appear in: , 2008 .

[16]  Eric Margolis,et al.  The Oxford handbook of philosophy of cognitive science , 2012 .

[17]  Daniel Rothschild,et al.  GAME THEORY AND SCALAR IMPLICATURES , 2013 .

[18]  Michael C. Frank,et al.  Learning and using language via recursive pragmatic reasoning about other agents , 2013, NIPS.

[19]  M. Lewis.  The role of pragmatic pressures in the emergence of linguistic structure , 2014 .

[20]  Noah D. Goodman,et al.  Formalizing the Pragmatics of Metaphor Understanding , 2014, CogSci.

[21]  Noah D. Goodman,et al.  Nonliteral understanding of number words , 2014, Proceedings of the National Academy of Sciences.

[22]  Naomi Feldman,et al.  Why discourse affects speakers’ choice of referring expressions , 2015, ACL.

[23]  Shimon Whiteson,et al.  Learning to Communicate with Deep Multi-Agent Reinforcement Learning , 2016, NIPS.

[24]  Shimon Whiteson,et al.  Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks , 2016, ArXiv.

[25]  Dan Klein,et al.  Reasoning about Pragmatics with Neural Listeners and Speakers , 2016, EMNLP.

[26]  Michael C. Frank,et al.  Review Pragmatic Language Interpretation as Probabilistic Inference , 2022 .

[27]  Pablo Hernandez-Leal,et al.  A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity , 2017, ArXiv.

[28]  Michael Franke,et al.  Game Theory in Pragmatics: Evolution, Rationality, and Reasoning , 2017 .

[29]  Ivan Titov,et al.  Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols , 2017, NIPS.

[30]  Gerard de Melo,et al.  Multimodal Question Answering over Structured Data with Ambiguous Entities , 2017, WWW.

[31]  Alexander Peysakhovich,et al.  Multi-Agent Cooperation and the Emergence of (Natural) Language , 2016, ICLR.

[32]  Christopher Potts,et al.  Pragmatically Informative Image Captioning with Character-Level Inference , 2018, NAACL.

[33]  Shimon Whiteson,et al.  Counterfactual Multi-Agent Policy Gradients , 2017, AAAI.

[34]  Nando de Freitas,et al.  Compositional Obverter Communication Learning From Raw Visual Input , 2018, ICLR.

[35]  Noah D. Goodman,et al.  Planning, Inference and Pragmatics in Sequential Language Games , 2018, TACL.

[36]  Stephen Clark,et al.  Emergent Communication through Negotiation , 2018, ICLR.

[37]  Peter Stone,et al.  Autonomous agents modelling other agents: A comprehensive survey and open problems , 2017, Artif. Intell..

[38]  Kyunghyun Cho,et al.  Emergent Communication in a Multi-Modal, Multi-Step Referential Game , 2017, ICLR.

[39]  Stephen Clark,et al.  Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input , 2018, ICLR.

[40]  Jason Lee,et al.  Emergent Translation in Multi-Agent Communication , 2017, ICLR.

[41]  Christopher Potts,et al.  An Incremental Iterated Response Model of Pragmatics , 2018, ArXiv.

[42]  Shimon Whiteson,et al.  QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning , 2018, ICML.

[43]  Joelle Pineau,et al.  On the Pitfalls of Measuring Emergent Communication , 2019, AAMAS.

[44]  Amanpreet Singh,et al.  Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks , 2018, ICLR.

[45]  Ellie Pavlick,et al.  Emergent Compositionality in Signaling Games , 2019, CogSci.

[46]  H. Francis Song,et al.  Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning , 2018, ICML.

[47]  Pierpaolo Battigalli,et al.  Incorporating belief-dependent motivation in games , 2019, Journal of Economic Behavior & Organization.

[48]  Gal Oren,et al.  Joint Optimization for Cooperative Image Captioning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[49]  Dan Klein,et al.  Pragmatically Informative Text Generation , 2019, NAACL.

[50]  Shimon Whiteson,et al.  The StarCraft Multi-Agent Challenge , 2019, AAMAS.

[51]  Leonidas J. Guibas,et al.  Shapeglot: Learning Language for Shape Differentiation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[52]  Michael Bowling,et al.  Ease-of-Teaching and Language Structure from Emergent Communication , 2019, NeurIPS.

[53]  Joelle Pineau,et al.  TarMAC: Targeted Multi-Agent Communication , 2018, ICML.

[54]  Mohamed Salah Zaïem,et al.  Learning to Communicate in Multi-Agent Reinforcement Learning : A Review , 2019, ArXiv.

[55]  Patrick Shafto,et al.  A mathematical theory of cooperative communication , 2019, NeurIPS.

[56]  Yafang Wang,et al.  Reinforcement Learning Over Knowledge Graphs for Explainable Dialogue Intent Mining , 2020, IEEE Access.

[57]  Chongjie Zhang,et al.  Learning Nearly Decomposable Value Functions Via Communication Minimization , 2019, ICLR.

[58]  Angeliki Lazaridou,et al.  Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning , 2020, ACL.

[59]  Roger P. Levy,et al.  A Rate-Distortion view of human pragmatic reasoning , 2020, ArXiv.