Markovian Decision Processes with Discrete Transition Law

We consider MDPs with countable state spaces and variable discount factors. The discount factor may depend on the state and the action. Under minimal assumptions we prove the reward iteration and formulate a structure theorem for MDPs. Also the useful notion of a bounding function is introduced.

[1]  J. Kiefer,et al.  The Inventory Problem: II. Case of Unknown Distributions of Demand , 1952 .

[2]  J. Kiefer,et al.  The Inventory Problem: I. Case of Known Distributions of Demand , 1952 .

[3]  Samuel Karlin,et al.  The structure of dynamic programing models , 1955 .

[4]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[5]  D. Blackwell Discrete Dynamic Programming , 1962 .

[6]  D. Blackwell Discounted Dynamic Programming , 1965 .

[7]  Onésimo Hernández-Lerma,et al.  Controlled Markov Processes , 1965 .

[8]  R. Bellman Dynamic programming. , 1957, Science.

[9]  George L. Nemhauser,et al.  Optimal capacity expansion , 1968 .

[10]  A. F. Veinott Discrete Dynamic Programming with Sensitive Discount Optimality Criteria , 1969 .

[11]  K. Hinderer,et al.  Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter , 1970 .

[12]  Karl Hinderer Instationäre dynamische Optimierung bei schwachen Voraussetzungen über die Gewinnfunktionen , 1971 .

[13]  Arie Hordijk,et al.  Dynamic programming and Markov potential theory , 1974 .

[14]  U. Rieder On stopped decision processes with discrete time parameter , 1975 .

[15]  S. Lippman On Dynamic Programming with Unbounded Rewards , 1975 .

[16]  K. Hinderer Estimates for finite-stage dynamic programs , 1976 .

[17]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Stochastic Control , 1977 .

[18]  K. Hinderer,et al.  An Improvement of J. F. Shapiro’s Turnpike Theorem for the Horizon of Finite Stage Discrete Dynamic Programs , 1977 .

[19]  G. Hübner,et al.  Bounds and good policies in stationary finite–stage Markovian decision problems , 1980, Advances in Applied Probability.

[20]  Sheldon M. Ross Introduction to Stochastic Dynamic Programming: Probability and Mathematical , 1983 .

[21]  M. K rn,et al.  Stochastic Optimal Control , 1988 .

[22]  O. Hernández-Lerma Adaptive Markov Control Processes , 1989 .

[23]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994, Wiley Series in Probability and Statistics.

[24]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control , 1995 .

[25]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[26]  Steven I. Marcus,et al.  Simulation-Based Algorithms for Markov Decision Processes , 2013 .

[27]  Anthony Almudevar,et al.  Approximate Iterative Algorithms , 2014 .