This paper is a continuation of a previous one which investigates programming over a Markov-renewal process---in which the intervals between transitions of a system from state i to state j are independent samples from a distribution that may depend upon both i and j. Given a reward structure, and a decision mechanism that influences both the rewards and the Markov-renewal process, the problem is to select alternatives at each transition so as to maximize total expected reward. The first portion of the paper investigated various finite-return models. In this part of the paper, we investigate the infinite-return models, where it becomes necessary to consider only stationary policies that maximize the dominant term in the reward. It is then important to specify whether the limiting experiment is I undiscounted, with the number of transitions n → ∞, II undiscounted, with a time horizon t → ∞, or III discounted, infinite n or t, with discount factor a → 0. In each case, a limiting form for the total expected reward is shown, and an algorithm developed to maximize the rate of return. The problem of finding the optimal or near-optimal policies in the case of ties is still computationally unresolved. Extensions to nonergodic processes are indicated, and special results for the two-state process are presented. Finally, an example of machine maintenance and repair is used to illustrate the generality of the models and the special problems that may arise.
[1]
R. Bellman.
A Markovian Decision Process
,
1957
.
[2]
William S. Jewell,et al.
The Properties of Recurrent-Event Processes
,
1960
.
[3]
A. S. Manne.
Linear Programming and Sequential Decisions
,
1960
.
[4]
Ronald A. Howard,et al.
Dynamic Programming and Markov Processes
,
1960
.
[5]
John G. Kemeny,et al.
Finite Markov Chains.
,
1960
.
[6]
R. Pyke.
Markov Renewal Processes with Finitely Many States
,
1961
.
[7]
R. Pyke.
Markov renewal processes: Definitions and preliminary properties
,
1961
.
[8]
David Blackwell,et al.
On the functional equation of dynamic programming
,
1961
.
[9]
W. Jewell.
MARKOV-RENEWAL PROGRAMMING
,
1962
.
[10]
M. Klein.
Inspection—Maintenance—Replacement Schedules Under Markovian Deterioration
,
1962
.
[11]
Stuart E. Dreyfus,et al.
Applied Dynamic Programming
,
1965
.
[12]
D. Blackwell.
Discrete Dynamic Programming
,
1962
.
[13]
G. Dantzig,et al.
Linear Programming in a Markov Chain
,
1962
.
[14]
C. Derman.
On Sequential Decisions and Markov Chains
,
1962
.
[15]
Ronald Pyke,et al.
Limit Theorems for Markov Renewal Processes
,
1964
.