Multi-Principal Assistance Games: Definition and Collegial Mechanisms

We introduce the concept of a multi-principal assistance game (MPAG), and circumvent an obstacle in social choice theory — Gibbard’s theorem — by using a sufficiently “collegial” preference inference mechanism. In an MPAG, a single agent assists N human principals who may have widely different preferences. MPAGs generalize assistance games, also known as cooperative inverse reinforcement learning games. We analyze in particular a generalization of apprenticeship learning in which the humans first perform some work to obtain utility and demonstrate their preferences, and then the robot acts to further maximize the sum of human payoffs. We show in this setting that if the game is sufficiently collegial — i.e., if the humans are responsible for obtaining a sufficient fraction of the rewards through their own actions — then their preferences are straightforwardly revealed through their work. This revelation mechanism is non-dictatorial, does not limit the possible outcomes to two alternatives, and is dominant-strategy incentive-compatible.

[1]  Anind K. Dey,et al.  Maximum Entropy Inverse Reinforcement Learning , 2008, AAAI.

[2]  A. Billard,et al.  Effects of repeated exposure to a humanoid robot on children with autism , 2004 .

[3]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[4]  Dritan Nace,et al.  Max-min fairness and its applications to routing and load-balancing in communication networks: a tutorial , 2008, IEEE Communications Surveys & Tutorials.

[5]  Allan Gibbard,et al.  Straightforwardness of Game Forms with Lotteries as Outcomes , 1978 .

[6]  Julie A. Shah,et al.  Fairness in Multi-Agent Sequential Decision-Making , 2014, NIPS.

[7]  Lawrence G. Sager.  Handbook of Computational Social Choice , 2015 .

[8]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[9]  Vincent Conitzer,et al.  The Revelation Principle for Mechanism Design with Reporting Costs , 2016, EC.

[10]  Craig Boutilier,et al.  Optimal social choice functions: A utilitarian view , 2015, Artif. Intell..

[11]  Dewen Hu,et al.  Multiobjective Reinforcement Learning: A Comprehensive Overview , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[12]  Chris L. Baker,et al.  Goal Inference as Inverse Planning , 2007 .

[13]  G. Thompson,et al.  The Theory of Committees and Elections. , 1959 .

[14]  Ariel D. Procaccia,et al.  Inverse Reinforcement Learning From Like-Minded Teachers , 2021, AAAI.

[15]  Shijian Li,et al.  Inverse Reinforcement Learning with Multiple Ranked Experts , 2019, ArXiv.

[16]  Takayuki Kanda,et al.  Interactive Robots as Social Partners and Peer Tutors for Children: A Field Trial , 2004, Hum. Comput. Interact..

[17]  S. Lalley,et al.  Quadratic Voting: How Mechanism Design Can Radicalize Democracy , 2017 .

[18]  Edith Elkind,et al.  Mechanism design: from partial to probabilistic verification , 2012, EC '12.

[19]  Karol Hausman,et al.  Learning to Interactively Learn and Assist , 2019, AAAI.

[20]  Anca D. Dragan,et al.  Cooperative Inverse Reinforcement Learning , 2016, NIPS.

[21]  Rudolf Kerschbamer,et al.  Do altruists lie less? , 2017, Journal of Economic Behavior & Organization.

[22]  Anca D. Dragan,et al.  Inverse Reward Design , 2017, NIPS.

[23]  Malte Jung,et al.  Reinforcement Learning with Fairness Constraints for Resource Distribution in Human-Robot Teams , 2019, ArXiv.

[24]  Eyal Amir,et al.  Bayesian Inverse Reinforcement Learning , 2007, IJCAI.

[25]  E. Fehr,et al.  Fairness and Retaliation: The Economics of Reciprocity , 2000, SSRN Electronic Journal.

[26]  Pierpaolo Battigalli,et al.  Deception: The role of guilt , 2013 .

[27]  A. Sen,et al.  Social Choice Theory , 1980 .

[28]  Andrew Y. Ng,et al.  Pharmacokinetics of a novel formulation of ivermectin after administration to goats , 2000, ICML.

[29]  Theodore Groves,et al.  Incentives in Teams , 1973 .

[30]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.