Graphical models for online solutions to interactive POMDPs

We develop a new graphical representation for interactive partially observable Markov decision processes (I-POMDPs) that is significantly more transparent and semantically clear than the previous representation. These graphical models called interactive dynamic influence diagrams (I-DIDs) seek to explicitly model the structure that is often present in real-world problems by decomposing the situation into chance and decision variables, and the dependencies between the variables. I-DIDs generalize DIDs, which may be viewed as graphical representations of POMDPs, to multiagent settings in the same way that I-POMDPs generalize POMDPs. I-DIDs may be used to compute the policy of an agent online as the agent acts and observes in a setting that is populated by other interacting agents. Using several examples, we show how I-DIDs may be applied and demonstrate their usefulness.

[1]  John C. Harsanyi,et al.  Games with Incomplete Information Played by "Bayesian" Players, I-III: Part I. The Basic Model& , 2004, Manag. Sci..

[2]  Drew Fudenberg,et al.  Game theory (3. pr.) , 1991 .

[3]  Ronald A. Howard,et al.  Readings on the Principles and Applications of Decision Analysis , 1989 .

[4]  Piotr J. Gmytrasiewicz,et al.  Learning models of other agents using influence diagrams , 1999 .

[5]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .

[6]  Piotr J. Gmytrasiewicz,et al.  Interactive dynamic influence diagrams , 2007, AAMAS '07.

[7]  Leslie Pack Kaelbling,et al.  Planning and Acting in Partially Observable Stochastic Domains , 1998, Artif. Intell..

[8]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[9]  Eddie Dekel,et al.  Hierarchies of Beliefs and Common Knowledge , 1993 .

[10]  Jonathan Schaeffer,et al.  The challenge of poker , 2002, Artif. Intell..

[11]  Robert J. Aumann,et al.  Interactive epistemology I: Knowledge , 1999, Int. J. Game Theory.

[12]  Edmund H. Durfee,et al.  Rational Coordination in Multi-Agent Environments , 2000, Autonomous Agents and Multi-Agent Systems.

[13]  Ross D. Shachter.  Evaluating Influence Diagrams , 1986, Oper. Res..

[14]  Prashant Doshi,et al.  Exact solutions of interactive POMDPs using behavioral equivalence , 2006, AAMAS '06.

[15]  Colin Camerer.  Behavioral Game Theory: Experiments in Strategic Interaction , 2003 .

[16]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[17]  Daphne Koller,et al.  Multi-Agent Influence Diagrams for Representing and Solving Games , 2001, IJCAI.

[18]  E. Fehr,et al.  Cooperation and Punishment in Public Goods Experiments , 1999, SSRN Electronic Journal.

[19]  P. J. Gmytrasiewicz,et al.  A Framework for Sequential Planning in Multi-Agent Settings , 2005, AI&M.

[20]  Ya'akov Gal,et al.  A language for modeling agents' decision making processes in games , 2003, AAMAS '03.