Means and Variances in Markov Reward Systems

In this paper, we study the total reward connected with a Markov reward process from time zero to time m. In particular, we determine the average reward within this time period, as well as its variance. Though the emphasis is on discrete time Markov processes, continuous-time reward processes will also be considered. For the discrete time reward process, the determination of the expected reward form 0 to m is of course trivial. It is of interest, however, that the deviation of this expectation from its steady state equivalent can be obtained from equations which are identical to the equations for the equilibrium probabilities, except that a vector of constants is added. We also consider the variance, both for transient systems and for systems in equilibrium. It is shown that the variance per time unit in equilibrium can also be obtained from equations which differ from the equations for the equilibrium probabilities only by a constant vector. Since there are three different sets of variables which satisfy similar equations, the LU factorization suggests itself. Instead of the LU factorization, we will use a UL factorization which reflects the probabilistic interpretation of the problem. This interpretation allows us to extend the factorization to systems with an infinite number of states as will be demonstrated, using the Wiener-Hopf factorization of the GI/G/1 queue as an example.

[1]  Winfried K. Grassmann,et al.  Numerical Solutions of the Waiting Time Distribution and Idle Time Distribution of the Arithmetic GI/G/1 Queue , 1989, Oper. Res..

[2]  W. K. Grassman,et al.  The Factorization of Queueing Equations and Their Interpretation , 1985 .

[3]  Winfried K. Grassmann,et al.  Regenerative Analysis and Steady State Distributions for Markov Chains , 1985, Oper. Res..

[4]  Donald Gross,et al.  The Randomization Technique as a Modeling Tool and Solution Procedure for Transient Markov Processes , 1984, Oper. Res..

[5]  Winfried K. Grassmann,et al.  Computation of Steady-State Probabilities for Infinite-State Markov Chains with Repeating Rows , 1993, INFORMS J. Comput..

[6]  Ward Whitt,et al.  Asymptotic Formulas for Markov Processes with Applications to Simulation , 1992, Oper. Res..

[7]  J. H. Jenkins,et al.  The relative efficiency of direct and maximum likelihood estimates of mean waiting time in the simple queue, M / M /1 , 1972 .

[8]  A. Jensen,et al.  Markoff chains as an aid in the study of Markoff processes , 1953 .

[9]  Winfried K. Grassmann Transient solutions in markovian queueing systems , 1977, Comput. Oper. Res..

[10]  Winfried K. Grassmann,et al.  Equilibrium distribution of block-structured Markov chains with repeating rows , 1990, Journal of Applied Probability.

[11]  J. Reynolds The covariance structure of queues and related processes – a survey of recent work , 1975, Advances in Applied Probability.

[12]  Winfried K. Grassmann,et al.  The asymptotic variance of a time average in a birth-death process , 1987 .

[13]  Winfried K. Grassmann,et al.  Means and variances of time averages in Markovian environments , 1987 .