Stochastic Games*

In a stochastic game the play proceeds by steps from position to position, according to transition probabilities controlled jointly by the two players. We shall assume a finite number, N , of positions, and finite numbers m K , n K of choices at each position; nevertheless, the game may not be bounded in length. If, when at position k , the players choose their i th and j th alternatives, respectively, then with probability s i j k > 0 the game stops, while with probability p i j k l the game moves to position l . Define s = min k , i , j s i j k . Since s is positive, the game ends with probability 1 after a finite number of steps, because, for any number t , the probability that it has not stopped after t steps is not more than (1 − s ) t . Payments accumulate throughout the course of play: the first player takes a i j k from the second whenever the pair i , j is chosen at position k. If we define the bound M: M = max k , i , j | a i j k | , then we see that the expected total gain or loss is bounded by M + ( 1 − s ) M + ( 1 − s ) 2 M + … = M / s . (1) The process therefore depends on N 2 + N matrices P K l = ( p i j k l | i = 1 , 2 , … , m K ; j = 1 , 2 , … , n K ) A K = ( a i j k | …