Some Monotonicity Results for Partially Observed Markov Decision Processes

This paper provides sufficient conditions for the optimal value in a discrete-time, finite, partially observed Markov decision process to be monotone on the space of state probability vectors ordered by likelihood ratios. The paper also presents sufficient conditions for the optimal policy to be monotone in a simple machine replacement problem, and, in the general case, for the optimal policy to be bounded from below by an easily calculated monotone function.

[1]  Ward Whitt,et al.  Comparison methods for queues and other stochastic models , 1986 .

[2]  William S. Lovejoy.  Ordered Solutions for Dynamic Programs , 1987, Math. Oper. Res..

[3]  G. Monahan.  State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms , 1982 .

[4]  C. White.  Monotone control laws for noisy, countable-state Markov chains , 1980 .

[5]  S. Ross.  Quality Control under Markovian Deterioration , 1971 .

[6]  C. White.  Optimal control-limit strategies for a partially observed replacement problem† , 1979 .

[7]  S. Karlin,et al.  Classes of orderings of measures and related correlation inequalities. I. Multivariate totally positive distributions , 1980 .

[8]  Karl Johan Åström,et al.  Optimal control of Markov processes with incomplete state information , 1965 .

[9]  D. Blackwell.  Discounted Dynamic Programming , 1965 .

[10]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Stochastic Control , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  W. Whitt.  Multivariate monotone likelihood ratio and uniform conditional stochastic order , 1982 .

[12]  W. Whitt.  A Note on the Influence of the Sample on the Posterior Distribution , 1979 .

[13]  Donald M. Topkis,et al.  Minimizing a Submodular Function on a Lattice , 1978, Oper. Res..

[14]  Donald B. Rosenfield,et al.  Markovian Deterioration with Uncertain Information , 1976, Oper. Res..

[15]  M. Aoki.  Optimal control of partially observable Markovian systems , 1965 .

[16]  S. Christian Albright,et al.  Structural Results for Partially Observable Markov Decision Processes , 1979, Oper. Res..

[17]  Edward J. Sondik,et al.  The optimal control of par-tially observable Markov processes , 1971 .

[18]  Edward J. Sondik,et al.  The Optimal Control of Partially Observable Markov Processes over a Finite Horizon , 1973, Oper. Res..