On Polynomial Sized MDP Succinct Policies

Policies of Markov Decision Processes (MDPs) determine the next action to execute from the current state and, possibly, the history (the past states). When the number of states is large, succinct representations are often used to compactly represent both the MDPs and the policies in a reduced amount of space. In this paper, some problems related to the size of succinctly represented policies are analyzed. Namely, it is shown that some MDPs have policies that can only be represented in space super-polynomial in the size of the MDP, unless the polynomial hierarchy collapses. This fact motivates the study of the problem of deciding whether a given MDP has a policy of a given size and reward. Since some algorithms for MDPs work by finding a succinct representation of the value function, the problem of deciding the existence of a succinct representation of a value function of a given size and reward is also considered.

[1]  Richard J. Lipton,et al.  Some connections between nonuniform and uniform complexity classes , 1980, STOC '80.

[2]  Craig Boutilier,et al.  Abstraction and Approximate Decision-Theoretic Planning , 1997, Artif. Intell..

[3]  Toniann Pitassi,et al.  Propositional Proof Complexity: Past, Present and Future , 2001, Bull. EATCS.

[4]  Daphne Koller,et al.  Computing Factored Value Functions for Policies in Structured MDPs , 1999, IJCAI.

[5]  MARTIN MUNDHENK.  The Complexity of Optimal Small Policies , 1999, Math. Oper. Res..

[6]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[7]  Paolo Liberatore.  Monotonic reductions, representative equivalence, and compilation of intractable problems , 2001, JACM.

[8]  Hans Tompits,et al.  Proof-complexity results for nonmonotonic reasoning , 2001, TOCL.

[9]  Larry J. Stockmeyer,et al.  The Polynomial-Time Hierarchy , 1976, Theor. Comput. Sci..

[10]  Keiji Kanazawa,et al.  A model for reasoning about persistence and causation , 1989 .

[11]  Michael L. Littman,et al.  The Computational Complexity of Probabilistic Planning , 1998, J. Artif. Intell. Res..

[12]  Eric Allender,et al.  Complexity of finite-horizon Markov decision process problems , 2000, JACM.

[13]  Michael L. Littman,et al.  Probabilistic Propositional Planning: Representations and Complexity , 1997, AAAI/IAAI.

[14]  Craig Boutilier,et al.  Decision-Theoretic Planning: Structural Assumptions and Computational Leverage , 1999, J. Artif. Intell. Res..

[15]  Zhengzhu Feng,et al.  Dynamic Programming for POMDPs Using a Factored State Representation , 2000, AIPS.

[16]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[17]  Michael L. Littman,et al.  Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes , 1997, UAI.

[18]  Paolo Liberatore,et al.  The size of MDP factored policies , 2002, AAAI/IAAI.

[19]  Alexander Russell,et al.  A Note on the Representational Incompatibility of Function Approximation and Factored Dynamics , 2002, NIPS.

[20]  J. Hartmanis,et al.  On the Computational Complexity of Algorithms , 1965 .

[21]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[22]  Ravi B. Boppana,et al.  The Complexity of Finite Functions , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[23]  Yishay Mansour,et al.  A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes , 1999, Machine Learning.

[24]  Weihong Zhang,et al.  Speeding Up the Convergence of Value Iteration in Partially Observable Markov Decision Processes , 2011, J. Artif. Intell. Res..

[25]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[26]  Francesco M. Donini,et al.  Preprocessing of Intractable Problems , 2002, Inf. Comput..

[27]  Anne Condon,et al.  On the Undecidability of Probabilistic Planning and Infinite-Horizon Partially Observable Markov Decision Problems , 1999, AAAI/IAAI.

[28]  David S. Johnson,et al.  A Catalog of Complexity Classes , 1991, Handbook of Theoretical Computer Science, Volume A: Algorithms and Complexity.

[29]  Leslie Pack Kaelbling,et al.  On the Complexity of Solving Markov Decision Problems , 1995, UAI.