Letter to the Editor - A Test for Suboptimal Actions in Markovian Decision Problems

In a Markovian decision problem, choice of an action determines an immediate return and the probability of moving to the next state. It is desired to maximize the expected total of discounted future returns. If upper and lower bounds on the optimal expected return are available, a simple test is described that may show that certain actions are suboptimal, permanently eliminating them from further consideration. This test may be incorporated into the dynamic programming routine for solving the decision problem. This was tried on Howard's automobile replacement problem, using the upper and lower bounds described in “A Modified Dynamic Programming Method” (J. Math. Anal. and Appl. 14, April, 1966). The amount of computation required by the dynamic programming routine was reduced, conservatively, by 75 per cent.