Technical Note - Identifying Forecast Horizons in Nonhomogeneous Markov Decision Processes

A procedure for identifying forecast horizons in nonhomogeneous Markov decision processes, based on convergence results for relative value functions, is developed. Two different algorithmic implementations of this procedure are discussed, and a closed form expression for computing sufficiently long horizons to guarantee epsilon optimality is presented.

[1]  Thomas E. Morton,et al.  Infinite-Horizon Dynamic Programming Models - A Planning-Horizon Formulation , 1979, Oper. Res..

[2]  D. White,et al.  Dynamic programming, Markov chains, and the method of successive approximations , 1963 .

[3]  Robert L. Smith,et al.  Conditions for the Existence of Planning Horizons , 1984, Math. Oper. Res..

[4]  Dean Isaacson,et al.  Markov Chains: Theory and Applications , 1976 .

[5]  M. Bartlett,et al.  Weak ergodicity in non-homogeneous Markov chains , 1958, Mathematical Proceedings of the Cambridge Philosophical Society.

[6]  Shinhong Kim,et al.  A Partially Observable Markov Decision Process with Lagged Information , 1987 .

[7]  Chelsea C. White,et al.  Parameter Imprecision in Finite State, Finite Action Dynamic Programs , 1986, Oper. Res..

[8]  Marius Iosifescu,et al.  Finite Markov Processes and Their Applications , 1981 .

[9]  Edward J. Sondik,et al.  The Optimal Control of Partially Observable Markov Processes over a Finite Horizon , 1973, Oper. Res..

[10]  A. Federgruen,et al.  The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms , 1978 .

[11]  A. Federgruen,et al.  A note on simultaneous recurrence conditions on a set of denumerable stochastic matrices : (preprint) , 1978 .

[12]  K. Hinderer,et al.  An Improvement of J. F. Shapiro’s Turnpike Theorem for the Horizon of Finite Stage Discrete Dynamic Programs , 1977 .

[13]  Robert L. Smith,et al.  A New Optimality Criterion for Nonhomogeneous Markov Decision Processes , 1987, Oper. Res..

[14]  Valerie Isham,et al.  Non‐Negative Matrices and Markov Chains , 1983 .

[15]  P. Kleindorfer,et al.  Stochastic Horizons for the Aggregate Planning Problem , 1978 .

[16]  T. Morton,et al.  Discounting, Ergodicity and Convergence for Markov Decision Processes , 1977 .

[17]  J. Shapiro.  Turnpike Planning Horizons for a Markovian Decision Model , 1968 .

[18]  Suresh Chand,et al.  Planning Horizon Procedures for Machine Replacement Models , 1979 .

[19]  T. Morton.  The Nonstationary Infinite Horizon Inventory Problem , 1978 .

[20]  R. Bellman.  Dynamic programming. , 1957, Science.