Encyclopaedia of Complexity Results for Finite-Horizon Markov Decision Process Problems

The computational complexity of finite horizon policy evaluation and policy existence problems are studied for several policy types and representations of Markov decision processes. In almost all cases, the problems are shown to be complete for their complexity classes; classes range from nondeterministic logarithmic space and probabilistic logarithmic space (highly parallelizable classes) to exponential space. In many cases, this work shows that problems that already were widely believed to be hard to compute are probably intractable (complete for NP, NP^PP , or PSPACE), or provably intractable (EXPTIME-complete or worse). The major contributions of the paper are to pinpoint the complexity of these problems; to isolate the factors that make these problems computationally complex; to show that even problems such as median-policy or average-policy evaluation may be intractable; and the introduction of natural NP^PP-complete problems.

[1]  F. d'Epenoux,et al.  A Probabilistic Production and Inventory Problem , 1963 .

[2]  Allan Borodin,et al.  Parallel Computation for Well-Endowed Rings and Space-Bounded Probabilistic Machines , 1984, Inf. Control..

[3]  György Turán,et al.  On the succinct representation of graphs , 1984, Discret. Appl. Math..

[4]  Hermann Jung.  On Probabilistic Time and Space , 1985, ICALP.

[5]  Christos Papadimitriou,et al.  Intractable problems in control theory , 1985, 1985 24th IEEE Conference on Decision and Control.

[6]  Mihalis Yannakakis,et al.  A Note on Succinct Representations of Graphs , 1986, Inf. Control..

[7]  David Chapman,et al.  Planning for Conjunctive Goals , 1987, Artif. Intell..

[8]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[9]  Richard E. Ladner.  Polynomial Space Counting Problems , 1989, SIAM J. Comput..

[10]  P. Tseng.  Solving H-horizon, stationary Markov decision problems in time proportional to log(H) , 1990 .

[11]  Daniel A. Spielman,et al.  PP is closed under intersection , 1991, STOC '91.

[12]  Seinosuke Toda,et al.  PP is as Hard as the Polynomial-Time Hierarchy , 1991, SIAM J. Comput..

[13]  J. Tor,et al.  Complexity classesde ned by counting quanti ers * , 1991 .

[14]  A. Stephen,et al.  Gap-De nable Counting Classes , 1991 .

[15]  W. Lovejoy.  A survey of algorithmic methods for partially observed Markov decision processes , 1991 .

[16]  V Vinay.  Counting auxiliary pushdown automata and semi-unbounded arithmetic circuits , 1991, [1991] Proceedings of the Sixth Annual Structure in Complexity Theory Conference.

[17]  Stephen A. Fenner,et al.  Gap-deenable Counting Classes , 1991 .

[18]  José L. Balcázar,et al.  The complexity of algorithmic problems on succinct instances , 1992 .

[19]  Carme Àlvarez,et al.  A Very Hard log-Space Counting Class , 1993, Theor. Comput. Sci..

[20]  Eric Allender,et al.  Relationships among PL, #L, and the determinant , 1994, Proceedings of IEEE 9th Annual Conference on Structure in Complexity Theory.

[21]  Tom Bylander,et al.  The Computational Complexity of Propositional STRIPS Planning , 1994, Artif. Intell..

[22]  Nicholas Kushmerick,et al.  An Algorithm for Probabilistic Planning , 1995, Artif. Intell..

[23]  V. S. Subrahmanian,et al.  Complexity, Decidability and Undecidability Results for Domain-Independent Planning , 1995, Artif. Intell..

[24]  Craig Boutilier,et al.  Exploiting Structure in Policy Construction , 1995, IJCAI.

[25]  Christer Bäckström,et al.  Expressive Equivalence of Planning Formalisms , 1995, Artif. Intell..

[26]  Michel de Rougemont,et al.  On the Complexity of Partially Observed Markov Decision Processes , 1996, Theor. Comput. Sci..

[27]  Craig Boutilier,et al.  Computing Optimal Policies for Partially Observable Decision Processes Using Compact Representations , 1996, AAAI/IAAI, Vol. 2.

[28]  Richard Washington,et al.  Incremental Markov-model planning , 1996, Proceedings Eighth IEEE International Conference on Tools with Artificial Intelligence.

[29]  Jos,et al.  The Complexity of Searching Implicit Graphs , 1996 .

[30]  Dana S. Nau,et al.  Complexity results for hierarchical task-network planning , 1996 .

[31]  T. Dean,et al.  Planning under uncertainty: structural assumptions and computational leverage , 1996 .

[32]  Eric Allender,et al.  The Complexity of Policy Evaluation for Finite-Horizon Partially-Observable Markov Decision Processes , 1997, MFCS.

[33]  Michael L. Littman,et al.  The Complexity of Plan Existence and Evaluation in Probabilistic Domains , 1997, UAI.

[34]  Michael L. Littman,et al.  Probabilistic Propositional Planning: Representations and Complexity , 1997, AAAI/IAAI.

[35]  Jacobo Torr.  Complexity Classes Deened by Counting Quantiiers* , 2022 .