Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria

A key challenge in the study of multiagent cooperation is the need for individual agents not only to cooperate effectively, but to decide with whom to cooperate. This is particularly critical in situations when other agents have hidden, possibly misaligned motivations and goals. Social deduction games offer an avenue to study how individuals might learn to synthesize potentially unreliable information about others, and elucidate their true motivations. In this work, we present Hidden Agenda, a two-team social deduction game that provides a 2D environment for studying learning agents in scenarios of unknown team alignment. The environment admits a rich set of strategies for both teams. Reinforcement learning agents trained in Hidden Agenda show that agents can learn a variety of behaviors, including partnering and voting without need for communication in natural language.

[1]  Thomas W. Malone,et al.  What makes things fun to learn? heuristics for designing instructional computer games , 1980, SIGSMALL '80.

[2]  C. Causer.  The Art of War , 2011, IEEE Potentials.

[3]  Joel Z. Leibo,et al.  Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot , 2021, ICML.

[4]  Chris L. Baker,et al.  Rational quantitative attribution of beliefs, desires and percepts in human mentalizing , 2017, Nature Human Behaviour.

[5]  Joel Z. Leibo,et al.  Multi-agent Reinforcement Learning in Sequential Social Dilemmas , 2017, AAMAS.

[6]  Joshua B. Tenenbaum,et al.  Finding Friend and Foe in Multi-Agent Games , 2019, NeurIPS.

[7]  Pujana Paliyawan,et al.  Player Dominance Adjustment: Promoting Self-Efficacy and Experience of Game Players by Adjusting Dominant Power , 2019, 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE).

[8]  Joel Z. Leibo,et al.  Open Problems in Cooperative AI , 2020, ArXiv.

[9]  Joshua B. Tenenbaum,et al.  The Naïve Utility Calculus: Computational Principles Underlying Commonsense Psychology , 2016, Trends in Cognitive Sciences.

[10]  Joel Z. Leibo,et al.  DeepMind Lab2D , 2020, ArXiv.

[11]  Yohei Nakata,et al.  Overview of AIWolfDial 2019 Shared Task: Contest of Automatic Dialog Agents to Play the Werewolf Game through Conversations , 2019 .

[12]  Joel Z. Leibo,et al.  OPtions as REsponses: Grounding behavioural hierarchies in multi-agent reinforcement learning , 2020, ICML.

[13]  Joel Z. Leibo,et al.  A multi-agent reinforcement learning model of common-pool resource appropriation , 2017, NIPS.

[14]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[15]  Angeliki Lazaridou,et al.  Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning , 2020, ACL.

[16]  C. D. De Dreu.  Human Cooperation , 2013, Psychological science in the public interest : a journal of the American Psychological Society.

[17]  Anca D. Dragan,et al.  On the Utility of Learning about Humans for Human-AI Coordination , 2019, NeurIPS.

[18]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[19]  Igor Mordatch,et al.  Emergent Tool Use From Multi-Agent Autocurricula , 2019, ICLR.

[20]  Joel Z. Leibo,et al.  Inequity aversion improves cooperation in intertemporal social dilemmas , 2018, NeurIPS.

[21]  Luca Iocchi,et al.  RLupus: Cooperation through emergent communication in The Werewolf social deduction game , 2021, ArXiv.

[22]  Tomoyuki Kaneko,et al.  Application of Deep Reinforcement Learning in Werewolf Game Agents , 2018, 2018 Conference on Technologies and Applications of Artificial Intelligence (TAAI).

[23]  Joel Z. Leibo,et al.  Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning , 2020, AAMAS.

[24]  Geoffrey Engelstein,et al.  Building Blocks of Tabletop Game Design , 2019 .

[25]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[26]  Jack Reinhardt,et al.  Competing in a Complex Hidden Role Game with Information Set Monte Carlo Tree Search , 2020, ArXiv.

[27]  Giampiero Salvi,et al.  A gaze-based method for relating group involvement to individual engagement in multimodal multiparty dialogue , 2013, ICMI '13.

[28]  Hayley Hung,et al.  The idiap wolf corpus: exploring group behaviour in a competitive role-playing game , 2010, ACM Multimedia.

[29]  Tom Schaul,et al.  Reinforcement Learning with Unsupervised Auxiliary Tasks , 2016, ICLR.

[30]  Filipe Wall Mutz,et al.  Hindsight policy gradients , 2017, ICLR.

[31]  Shane Legg,et al.  IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures , 2018, ICML.

[32]  Derek Thompson,et al.  Teaching Validity and Soundness of Arguments Using the Board Game: The Resistance , 2015 .

[33]  Scott E. Hudson,et al.  Social Group Interactions in a Role-Playing Game , 2015, HRI.

[34]  J. H. Davis,et al.  The Social Psychology of Small Groups: Cooperative and Mixed-Motive Interaction , 1976 .

[35]  Chris Martens,et al.  A Study of AI Agent Commitment in One Night Ultimate Werewolf with Human Players , 2019, AIIDE.