Survey of linear programming for standard and nonstandard Markovian control problems. Part II: Applications

This paper deals with some applications of Markov decision models for which the linear programming method is efficient. These models are replacement models (with the optimal stopping problem as special case), separable models (including the inventory model as special case) and the multi-armed bandit model. In the companion paper “Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory”, general linear programming methods are discussed. These linear programming formulations are the starting point for the efficient methods that will be derived for the special models.

[1]  E. Denardo.  Separable Markovian Decision Problems , 1968 .

[2]  Matthew J. Sobel,et al.  Myopic Solutions of Markov Decision Processes and Stochastic Games , 1981, Oper. Res..

[3]  J. Gittins.  Bandit processes and dynamic allocation indices , 1979 .

[4]  L. Kallenberg.  Separable Markovian decision problems , 1992 .

[5]  P. Whittle.  Multi‐Armed Bandits and the Gittins Index , 1980 .

[6]  Michael N. Katehakis,et al.  Linear Programming for Finite State Multi-Armed Bandit Problems , 1986, Math. Oper. Res..

[7]  Stef Tijs,et al.  Stochastic games with state independent transitions and separable rewards , 1984 .

[8]  L. C. M. Kallenberg,et al.  Linear programming and finite Markovian control problems , 1984 .

[9]  Jean Walrand,et al.  Extensions of the multiarmed bandit problem: The discounted case , 1985 .

[10]  R. Weber.  On the Gittins Index for Multiarmed Bandits , 1992 .

[11]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[12]  Lodewijk C. M. Kallenberg,et al.  A Note on M. N. Katehakis' and Y.-R. Chen's Computation of the Gittins Index , 1986, Math. Oper. Res..

[13]  G. D. Eppen,et al.  Linear Programming Solutions for Separable Markovian Decision Problems , 1967 .

[14]  Michael N. Katehakis,et al.  The Multi-Armed Bandit Problem: Decomposition and Computation , 1987, Math. Oper. Res..

[15]  Lodewijk C. M. Kallenberg,et al.  Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory , 1994, Math. Methods Oper. Res..