Markov Strategies are Better than stationary Strategies

We examine the use of stationary and Markov strategies in zero-sum stochastic games with finite state and action spaces. It is natural to evaluate a strategy for the maximising player, player 1, by the highest reward guaranteed to him against any strategy of the opponent. The highest rewards guaranteed by stationary strategies or by Markov strategies are called the stationary utility or the Markov utility, respectively. Since all stationary strategies are Markov strategies, the Markov utility is always larger or equal to the stationary utility. However, in all presently known subclasses of stochastic games, these utilities turn out to be equal. In this paper, we provide a colourful example in which the Markov utility is strictly larger than the stationary utility and we present several conditions under which the utilities are equal. We also show that each stochastic game has at least one initial state for which the two utilities are equal. Several examples clarify these issues.

[1]  Stef Tijs,et al.  Stochastic games with state independent transitions and separable rewards , 1984 .

[2]  John Cubbin,et al.  Optimality and Equilibria in Stochastic Games , 1994 .

[3]  Arie Hordijk,et al.  Semi-markov strategies in stochastic games , 1983 .

[4]  A. Neyman,et al.  Stochastic games , 1981 .

[5]  A. W. Tucker,et al.  Contributions to the Theory of Games, Vol. II , 1954 .

[6]  E. Kohlberg.  Repeated Games with Absorbing States , 1974 .

[7]  O. J. Vrieze,et al.  Easy initial states in stochastic games , 1991 .

[8]  M. J. Sobel.  Noncooperative Stochastic Games , 1971 .

[10]  J. Filar.  Ordered field property for stochastic games when the player who controls transitions changes from state to state , 1981 .

[11]  Koos Vrieze,et al.  Stationary ɛ-optimal strategies in stochastic games , 1993 .

[12]  D. Blackwell,et al.  THE BIG MATCH , 1968, Classics in Game Theory.

[13]  O. J. Vrieze,et al.  Simplifying Optimal Strategies in Stochastic Games , 1998 .

[14]  T. E. S. Raghavan,et al.  Positive Stochastic Games and a Theorem of Ornstein , 1991 .

[15]  S. Lippman,et al.  Stochastic Games with Perfect Information and Time Average Payoff , 1969 .

[16]  Frank Thuijsman.  Non-zerosum stochastic games , 1987 .

[17]  J. M. Coulomb.  Repeated games with absorbing states and no signals , 1992 .

[18]  Dean Gillette,et al.  9. STOCHASTIC GAMES WITH ZERO STOP PROBABILITIES , 1958 .

[19]  O. J. Vrieze,et al.  On stochastic games with additive reward and transition structure , 1985 .