Minimizing Expected Termination Time in One-Counter Markov Decision Processes

We consider the problem of computing the value and an optimal strategy for minimizing the expected termination time in one-counter Markov decision processes. Since the value may be irrational and an optimal strategy may be rather complicated, we concentrate on the problems of approximating the value up to a given error e>0 and computing a finite representation of an e-optimal strategy. We show that these problems are solvable in exponential time for a given configuration, and we also show that they are computationally hard in the sense that a polynomial-time approximation algorithm cannot exist unless P=NP.

[1]  Kousha Etessami,et al.  One-Counter Stochastic Games , 2010, FSTTCS.

[2]  Kousha Etessami,et al.  Recursive Markov Decision Processes and Recursive Stochastic Games , 2005, ICALP.

[3]  Jeffrey Shallit,et al.  Algorithmic Number Theory , 1996, Lecture Notes in Computer Science.

[4]  Olivier Serre,et al.  Parity Games Played on Transition Graphs of One-Counter Processes , 2006, FoSSaCS.

[5]  Kousha Etessami,et al.  Recursive Stochastic Games with Positive Rewards , 2008, ICALP.

[6]  S. Br,et al.  QUALITATIVE REACHABILITY IN STOCHASTIC BPA GAMES , 2009 .

[7]  Kousha Etessami,et al.  Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems , 2008, QEST.

[8]  Antonín Kucera,et al.  The complexity of bisimilarity-checking for one-counter processes , 2003, Theor. Comput. Sci..

[9]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[10]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[11]  Krishnendu Chatterjee,et al.  Generalized Mean-payoff and Energy Games , 2010, FSTTCS.

[12]  Petr Jancar,et al.  A note on emptiness for alternating finite automata with a one-letter alphabet , 2007, Inf. Process. Lett..

[13]  David Williams,et al.  Probability with Martingales , 1991, Cambridge mathematical textbooks.

[14]  Tomás Brázdil,et al.  Qualitative reachability in stochastic BPA games , 2011, Inf. Comput..

[15]  Vaidyanathan Ramaswami,et al.  Introduction to Matrix Analytic Methods in Stochastic Modeling , 1999, ASA-SIAM Series on Statistics and Applied Mathematics.

[16]  Tomás Brázdil,et al.  Reachability in recursive Markov decision processes , 2008, Inf. Comput..

[17]  Faron Moller,et al.  DP lower bounds for equivalence-checking and model-checking of one-counter automata , 2004, Inf. Comput..

[18]  Krishnendu Chatterjee,et al.  Energy Parity Games , 2010, ICALP.

[19]  Kousha Etessami,et al.  Approximating the Termination Value of One-Counter MDPs and Stochastic Games , 2011, ICALP.

[20]  J. Filar,et al.  Competitive Markov Decision Processes , 1996 .

[21]  Kousha Etessami,et al.  One-counter Markov decision processes , 2009, SODA '10.

[22]  Kousha Etessami,et al.  Efficient Qualitative Analysis of Classes of Recursive Markov Decision Processes and Simple Stochastic Games , 2006, STACS.

[23]  Kousha Etessami,et al.  Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems , 2008, 2008 Fifth International Conference on Quantitative Evaluation of Systems.

[24]  Tomás Brázdil,et al.  Efficient Analysis of Probabilistic Programs with an Unbounded Counter , 2011, CAV.