Factored Markov decision processes with Imprecise Transition Probabilities

This paper presents a short survey of the research we have carried out on planning under uncertainty where we consider different forms of imprecision on the probability transition functions. Our main results are on efficient solutions for Markov Decision Process with Imprecise Transition Probabilities (MDP-IPs), a generalization of a Markov Decision Process where the imprecise probabilities are given in terms of credal sets. Noting that the key computational bottleneck in the solution of MDP-IPs is the need to repeatedly solve an optimization problem. Our results show how to target approximation techniques to drastically reduce the computational overhead of the optimization solver while producing bounded, approximately optimal solutions.

[1]  Chelsea C. White,et al.  Markov Decision Processes with Imprecise Transition Probabilities , 1994, Oper. Res..

[2]  Jigui Sun,et al.  Solving Uncertain Markov Decision Problems: An Interval-Based Method , 2006, ICNC.

[3]  Nevin L. Zhang,et al.  A simple approach to Bayesian network computations , 1994 .

[4]  Olivier Buffet,et al.  Robust Planning with (L)RTDP , 2005, IJCAI.

[5]  P. Walley.  Statistical Reasoning with Imprecise Probabilities , 1990 .

[6]  J. K. Satia,et al.  Markovian Decision Processes with Uncertain Transition Probabilities , 1973, Oper. Res..

[7]  Craig Boutilier,et al.  Context-Specific Independence in Bayesian Networks , 1996, UAI.

[8]  Ricardo Shirota Filho,et al.  Multilinear and integer programming for markov decision processes with imprecise probabilities , 2007 .

[9]  Robert Givan,et al.  Bounded-parameter Markov decision processes , 2000, Artif. Intell..

[10]  Minghao Yin,et al.  Solving Planning Under Uncertainty: Quantitative and Qualitative Approach , 2007, IFSA.

[11]  Jesse Hoey,et al.  APRICODD: Approximate Policy Construction Using Decision Diagrams , 2000, NIPS.

[12]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[13]  Fabio Gagliardi Cozman,et al.  Credal networks , 2000, Artif. Intell..

[14]  Scott Sanner,et al.  Using mathematical programming to solve Factored Markov Decision Processes with Imprecise Probabilities , 2011, Int. J. Approx. Reason..

[15]  R. I. Bahar,et al.  Algebraic decision diagrams and their applications , 1993, Proceedings of 1993 International Conference on Computer Aided Design (ICCAD).

[16]  Jesse Hoey,et al.  SPUDD: Stochastic Planning using Decision Diagrams , 1999, UAI.

[17]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[18]  Fabio Gagliardi Cozman,et al.  Graphical models for imprecise probabilities , 2005, Int. J. Approx. Reason..

[19]  Keiji Kanazawa,et al.  A model for reasoning about persistence and causation , 1989 .

[20]  Shobha Venkataraman,et al.  Efficient Solution Algorithms for Factored MDPs , 2003, J. Artif. Intell. Res..

[21]  F. Cozman,et al.  Representing and solving factored markov decision processes with imprecise probabilities , 2009 .

[22]  Laurent El Ghaoui,et al.  Robust Control of Markov Decision Processes with Uncertain Transition Matrices , 2005, Oper. Res..

[23]  Craig Boutilier,et al.  Decision-Theoretic Planning: Structural Assumptions and Computational Leverage , 1999, J. Artif. Intell. Res..

[24]  Fabio Gagliardi Cozman,et al.  Planning under Risk and Knightian Uncertainty , 2007, IJCAI.

[25]  Scott Sanner,et al.  Solutions to Factored MDPs with Imprecise Transition Probabilities 1 , 2011 .