On the existence of stationary optimal strategies

The question with which this paper is concerned is roughly speaking: In a gambling situation or dynamical programming situation, are strategies that take the past into account any better than those that are based only on the present situation? Let us now state precisely what situations we will be dealing with. We have a set X, the states of our system (e.g. how much money you have). For each state xCX we have a collection V: of gambles available to you when in state x. Each gamble v E V: will be a measure on X with support on a countable number of points. (If you chose v you go from x to y with probability v(y).) A strategy tells you how to choose a gamble, vn, on the nth day as a function of the previous history of the system (xl, x2, . * *, xn) (vn must be in Vn ).1 A stationary strategy is one where the choice of vn depends only on x". Suppose there is a special state g(our goal). Then for each strategy s we have a function on X, F8(x) the probability of reaching g if we start at x and use s. Let F(x) =sup F8 (x), where sup is taken over all strategies s. Our main result is