Markovian Decision Processes with Discrete Transition Law

We consider MDPs with countable state spaces and variable discount factors. The discount factor may depend on the state and the action. Under minimal assumptions we prove the reward iteration and formulate a structure theorem for MDPs. Also the useful notion of a bounding function is introduced.

[1]  J. Kiefer,et al.  The Inventory Problem: II. Case of Unknown Distributions of Demand , 1952 .

[2]  J. Kiefer,et al.  The Inventory Problem: I. Case of Known Distributions of Demand , 1952 .

[3]  Samuel Karlin,et al.  The structure of dynamic programing models , 1955 .

[4]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[5]  D. Blackwell.  Discrete Dynamic Programming , 1962 .

[6]  D. Blackwell.  Discounted Dynamic Programming , 1965 .

[7]  Onésimo Hernández-Lerma,et al.  Controlled Markov Processes , 1965 .

[8]  R. Bellman.  Dynamic programming. , 1957, Science.

[9]  George L. Nemhauser,et al.  Optimal capacity expansion , 1968 .

[10]  A. F. Veinott.  Discrete Dynamic Programming with Sensitive Discount Optimality Criteria , 1969 .

[11]  K. Hinderer,et al.  Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter , 1970 .

[12]  Karl Hinderer.  Instationäre dynamische Optimierung bei schwachen Voraussetzungen über die Gewinnfunktionen , 1971 .

[13]  Arie Hordijk,et al.  Dynamic programming and Markov potential theory , 1974 .

[14]  U. Rieder.  On stopped decision processes with discrete time parameter , 1975 .

[15]  S. Lippman.  On Dynamic Programming with Unbounded Rewards , 1975 .

[16]  K. Hinderer.  Estimates for finite-stage dynamic programs , 1976 .

[17]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Stochastic Control , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[18]  K. Hinderer,et al.  An Improvement of J. F. Shapiro’s Turnpike Theorem for the Horizon of Finite Stage Discrete Dynamic Programs , 1977 .

[19]  G. Hübner,et al.  Bounds and good policies in stationary finite–stage Markovian decision problems , 1980, Advances in Applied Probability.

[20]  Sheldon M. Ross,et al.  Introduction to Stochastic Dynamic Programming: Probability and Mathematical , 1983 .

[21]  M. K rn,et al.  Stochastic Optimal Control , 1988 .

[22]  O. Hernández-Lerma.  Adaptive Markov Control Processes , 1989 .

[23]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[24]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[25]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[26]  Steven I. Marcus,et al.  Simulation-based Algorithms for Markov Decision Processes/ Hyeong Soo Chang ... [et al.] , 2013 .

[27]  Anthony Almudevar,et al.  Approximate Iterative Algorithms , 2014 .