Learning Reciprocity in Complex Sequential Social Dilemmas

Reciprocity is an important feature of human social interaction and underpins our cooperative nature. What is more, simple forms of reciprocity have proved remarkably resilient in matrix game social dilemmas. Most famously, the tit-for-tat strategy performs very well in tournaments of Prisoner's Dilemma. Unfortunately this strategy is not readily applicable to the real world, in which options to cooperate or defect are temporally and spatially extended. Here, we present a general online reinforcement learning algorithm that displays reciprocal behavior towards its co-players. We show that it can induce pro-social outcomes for the wider group when learning alongside selfish agents, both in a $2$-player Markov game, and in $5$-player intertemporal social dilemmas. We analyse the resulting policies to show that the reciprocating agents are strongly influenced by their co-players' behavior.

[1]  L. Shapley,et al.  Stochastic Games* , 1953, Proceedings of the National Academy of Sciences.

[2]  P. Blau.  Exchange and Power in Social Life , 1964 .

[3]  M. Olson,et al.  The Logic of Collective Action , 1965 .

[4]  G. Hardin,et al.  The Tragedy of the Commons , 1968, Green Planet Blues.

[5]  C. Granger.  Investigating Causal Relations by Econometric Models and Cross-Spectral Methods , 1969 .

[6]  A. Rapoport,et al.  Prisoner's Dilemma: A Study in Conflict and Co-operation , 1970 .

[7]  R. Trivers.  The Evolution of Reciprocal Altruism , 1971, The Quarterly Review of Biology.

[8]  R. Axelrod.  More Effective Choice in the Prisoner's Dilemma , 1980 .

[9]  W. Hamilton,et al.  The evolution of cooperation. , 1984, Science.

[10]  B. Latané.  The psychology of social impact. , 1981 .

[11]  T. L. Schwartz.  The Logic of Collective Action , 1986 .

[12]  R. Boyd.  Mistakes allow evolutionary stability in the repeated prisoner's dilemma game. , 1989, Journal of theoretical biology.

[13]  Robin I. M. Dunbar.  Coevolution of neocortical size, group size and language in humans , 1993, Behavioral and Brain Sciences.

[14]  G. Brady.  Governing the Commons: The Evolution of Institutions for Collective Action , 1993 .

[15]  M. Nowak,et al.  A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner's Dilemma game , 1993, Nature.

[16]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[17]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[18]  E. Ostrom.  A Behavioral Approach to the Rational Choice Theory of Collective Action: Presidential Address, American Political Science Association, 1997 , 1998, American Political Science Review.

[19]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[20]  Kagan Tumer,et al.  An Introduction to Collective Intelligence , 1999, ArXiv.

[21]  C. Granger.  Investigating causal relations by econometric models and cross-spectral methods , 1969 .

[22]  K. Laland,et al.  Social intelligence, innovation, and enhanced brain size in primates , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[23]  T. Chartrand,et al.  The Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconscious Mimicry , 2003 .

[24]  Nuttapong Chentanez,et al.  Intrinsically Motivated Reinforcement Learning , 2004, NIPS.

[25]  R. Axelrod,et al.  Evolutionary Dynamics , 2004 .

[26]  Enrique Fatás Juberías,et al.  Reciprocity, matching and conditional Cooperation in two public goods games , 2005 .

[27]  Noah J. Goldstein,et al.  Social influence: compliance and conformity. , 2004, Annual review of psychology.

[28]  N. Bardsley,et al.  Conformity and reciprocity in public good provision , 2005 .

[29]  R. Johnstone,et al.  Indirect reciprocity in asymmetric interactions: when apparent altruism facilitates profitable exploitation , 2007, Proceedings of the Royal Society B: Biological Sciences.

[30]  J. Bendor,et al.  Effective Choice in the Prisoner ' s Dilemma , 2007 .

[31]  Sandip Sen,et al.  Emergence of Norms through Social Learning , 2007, IJCAI.

[32]  Robin J. Tanner,et al.  Of chameleons and consumption: The impact of mimicry on choice and preferences. , 2008 .

[33]  Robert L. Goldstone,et al.  Effect of rule choice in dynamic interactive spatial commons , 2008 .

[34]  Jason P. Mitchell.  Inferences about mental states , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[35]  T. Chartrand,et al.  The antecedents and consequences of human behavioral mimicry. , 2013, Annual review of psychology.

[36]  Peter Duersch,et al.  When is tit-for-tat unbeatable? , 2013, Int. J. Game Theory.

[37]  J. Henrich,et al.  The Big Man Mechanism: how prestige fosters cooperation and creates prosocial leaders , 2015, Philosophical Transactions of the Royal Society B: Biological Sciences.

[38]  S. Brandl,et al.  Coordinated vigilance provides evidence for direct reciprocity in coral reef fishes , 2015, Scientific Reports.

[39]  Joshua B. Tenenbaum,et al.  Coordinate to cooperate or compete: Abstract goals and joint intentions in social interaction , 2016, CogSci.

[40]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[41]  A. Hamilton,et al.  Cognitive mechanisms for responding to mimicry from others , 2016, Neuroscience & Biobehavioral Reviews.

[42]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[43]  Mohamed Medhat Gaber,et al.  Imitation Learning , 2017, ACM Comput. Surv..

[44]  Alexander Peysakhovich,et al.  Maintaining cooperation in complex social dilemmas using deep reinforcement learning , 2017, ArXiv.

[45]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[46]  Joel Z. Leibo,et al.  A multi-agent reinforcement learning model of common-pool resource appropriation , 2017, NIPS.

[47]  Denis Gifford,et al.  1953 , 2018, The British Film Catalogue.

[48]  Joel Z. Leibo,et al.  Inequity aversion improves cooperation in intertemporal social dilemmas , 2018, NeurIPS.

[49]  S. Hewitt,et al.  2007 , 2018, Los 25 años de la OMC: Una retrospectiva fotográfica.

[50]  Alexander Peysakhovich,et al.  Consequentialist conditional cooperation in social dilemmas with imperfect information , 2017, AAAI Workshops.

[51]  S. Gächter,et al.  Leaders as role models and ‘belief managers’ in social dilemmas , 2018, Journal of Economic Behavior & Organization.

[52]  Alexander Peysakhovich,et al.  Learning Existing Social Conventions in Markov Games , 2018, 1806.10071.

[53]  Shane Legg,et al.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures , 2018, ICML.

[54]  Joel Z. Leibo,et al.  Inequity aversion resolves intertemporal social dilemmas , 2018, ArXiv.

[55]  S. Hewitt,et al.  2008 , 2018, Los 25 años de la OMC: Una retrospectiva fotográfica.

[56]  Guy Lever,et al.  Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward , 2018, AAMAS.

[57]  Denis Gifford.  1969 , 2018, The British Film Catalogue.

[58]  2018 , 2019, Communications of the ACM.

[59]  Marc Mézard,et al.  1993 , 1993, The Winning Cars of the Indianapolis 500.

[60]  H. Francis Song,et al.  Relational Forward Models for Multi-Agent Learning , 2018, ICLR.

[61]  Nando de Freitas,et al.  Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning , 2018, ICML.