论文信息 - 2 Büchi Objectives in Countable MDPs 1 Introduction

2 Büchi Objectives in Countable MDPs 1 Introduction

We study countably infinite Markov decision processes with Büchi objectives, which ask to visit a given subset F of states infinitely often. A question left open by T.P. Hill in 1979 [10] is whether there always exist ε-optimal Markov strategies, i.e., strategies that base decisions only on the current state and the number of steps taken so far. We provide a negative answer to this question by constructing a non-trivial counterexample. On the other hand, we show that Markov strategies with only 1 bit of extra memory are sufficient. 2012 ACM Subject Classification Theory of computation → Random walks and Markov chains; Mathematics of computing → Probability and statistics

C. Baier | S. Leonardi | P. Flocchini | Chatzigiannakis

[1] D. Ornstein. On the existence of stationary optimal strategies , 1969 .

[2] T. Hill. On the Existence of Good Markov Strategies , 1979 .

[3] Martin L. Puterman,et al. Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[4] Alex M. Andrew,et al. ROBOT LEARNING, edited by Jonathan H. Connell and Sridhar Mahadevan, Kluwer, Boston, 1993/1997, xii+240 pp., ISBN 0-7923-9365-1 (Hardback, 218.00 Guilders, $120.00, £89.95). , 1999, Robotica (Cambridge. Print).

[5] Manfred Schäl,et al. Markov Decision Processes in Finance and Dynamic Options , 2002 .

[6] Pieter Abbeel,et al. Learning first-order Markov models for control , 2004, NIPS.

[7] Theodore P. Hill,et al. goal problems in gambling theory , 2009 .

[8] J. Krcál,et al. Determinacy and Optimal Strategies in Stochastic Games , 2009 .

[9] Olivier Buffet,et al. Markov Decision Processes in Artificial Intelligence , 2010 .

[10] U. Rieder,et al. Markov Decision Processes with Applications to Finance , 2011 .

[11] Krishnendu Chatterjee,et al. A survey of stochastic ω-regular games , 2012, J. Comput. Syst. Sci..

[12] Stefan Kiefer,et al. Parity objectives in countable MDPs , 2017, 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).