Series Expansions for Continuous-Time Markov Processes

We present update formulas that allow us to express the stationary distribution of a continuous-time Markov process with denumerable state space having generator matrix Q* through a continuous-time Markov process with generator matrix Q. Under suitable stability conditions, numerical approximations can be derived from the update formulas, and we show that the algorithms converge at a geometric rate. Applications to sensitivity analysis and bounds on perturbations are discussed as well. Numerical examples are presented to illustrate the efficiency of the proposed algorithm.

[1]  Rommert Dekker,et al.  On the Relation Between Recurrence and Ergodicity Properties in Denumerable Markov Decision Chains , 1994, Math. Oper. Res..

[2]  C. D. Meyer,et al.  Comparison of perturbation bounds for the stationary distribution of a Markov chain , 2001 .

[3]  E. Altman,et al.  Perturbation analysis for denumerable Markov chains with application to queueing models , 2004, Advances in Applied Probability.

[4]  A. Hordijk,et al.  SERIES EXPANSIONS FOR FINITE-STATE MARKOV CHAINS , 2007, Probability in the Engineering and Informational Sciences.

[5]  A. Yu. Mitrophanov,et al.  Stability Estimates for Finite Homogeneous Continuous-time Markov Chains , 2006 .

[6]  Yukio Takahashi,et al.  ON THE EFFECTS OF SMALL DEVIATIONS IN THE TRANSITION MATRIX OF A FINITE MARKOV CHAIN , 1973 .

[7]  Arie Hordijk,et al.  Characterization and sufficient conditions for normed ergodicity of Markov chains , 2004, Advances in Applied Probability.

[8]  S. Kirkland.  Conditioning properties of the stationary distribution for a Markov chain , 2003 .

[9]  P. Schweitzer.  Perturbation theory and finite Markov chains , 1968 .

[10]  Vidyadhar G. Kulkarni,et al.  Retrial queues revisited , 1998 .

[11]  Lasse Leskelä,et al.  Stabilization of an overloaded queueing network using measurement-based admission control , 2006 .

[12]  P. K. Pollett,et al.  Similar Markov chains , 2001, Journal of Applied Probability.

[13]  A. Hordijk,et al.  On ergodicity and recurrence properties of a Markov chain by an application to an open jackson network , 1992, Advances in Applied Probability.

[14]  Avishai Mandelbaum,et al.  Queueing Models of Call Centers: An Introduction , 2002, Ann. Oper. Res..

[15]  N. Vieille,et al.  Perturbed Markov chains , 2003, Journal of Applied Probability.

[16]  Xi-Ren Cao,et al.  Perturbation realization, potentials, and sensitivity analysis of Markov processes , 1997, IEEE Trans. Autom. Control..

[17]  Mark Borodovsky,et al.  SENSITIVITY OF HIDDEN MARKOV MODELS , 2005 .

[18]  Mark Borodovsky,et al.  Statistical significance in biological sequence analysis , 2006, Briefings Bioinform..

[19]  R. Tweedie.  Perturbations of countable Markov chains and processes , 1980, Advances in Applied Probability.

[20]  Avishai Mandelbaum,et al.  Telephone Call Centers: Tutorial, Review, and Research Prospects , 2003, Manuf. Serv. Oper. Manag..

[21]  A. Y. Mitrophanov,et al.  The spectral gap and perturbation bounds for reversible continuous-time Markov chains , 2004 .

[22]  Frank Kelly,et al.  Reversibility and Stochastic Networks , 1979 .

[23]  Arie Hordijk,et al.  Blackwell optimality in the class of all policies in Markov decision chains with a Borel state space and unbounded rewards , 1999, Math. Methods Oper. Res..

[24]  N. Kartashov,et al.  Strongly stable Markov chains , 1986 .

[25]  Qing Zhang,et al.  Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states , 2000 .

[26]  Antonio Di Crescenzo.  On certain transformation properties of birth-and-death processes , 1994 .

[27]  Arie Hordijk,et al.  Series Expansions for Finite-State Markov Chains , 2005 .

[28]  R. B. Lenin,et al.  Families of birth-death processes with similar time-dependent behaviour , 2000, Journal of Applied Probability.

[29]  Arie Hordijk,et al.  Average, Sensitive and Blackwell Optimal Policies in Denumerable Markov Decision Chains with Unbounded Rewards , 1988, Math. Oper. Res..

[30]  S. Lippman.  On Dynamic Programming with Unbounded Rewards , 1975 .

[31]  Xi-Ren Cao.  The Maclaurin series for performance functions of Markov chains , 1998, Advances in Applied Probability.

[32]  Tony Shardlow,et al.  A Perturbation Theory for Ergodic Markov Chains and Application to Numerical Approximations , 2000, SIAM J. Numer. Anal..

[33]  J. Templeton.  Retrial queues , 1999 .

[34]  A. Hordijk,et al.  Taylor series expansions for stationary Markov chains , 2003, Advances in Applied Probability.

[35]  Gennadi Falin,et al.  A survey of retrial queues , 1990, Queueing Syst. Theory Appl..

[36]  Antonio Di Crescenzo,et al.  On some transformations of bilateral birth-and-death processes with applications , 2008, 0803.1413.

[37]  Tony Shardlow,et al.  A perturbation theory for ergodic properties of Markov chains , 1998 .

[38]  Jesús R. Artalejo,et al.  Retrial Queueing Systems , 2008 .

[39]  A. Yu. Mitrophanov.  Ergodicity coefficient and perturbation bounds for continuous-time Markov chains , 2005 .

[40]  J. Kemeny,et al.  Potentials for denumerable Markov chains , 1961 .

[41]  Pauline Coolen-Schrijner,et al.  THE DEVIATION MATRIX OF A CONTINUOUS-TIME MARKOV CHAIN , 2002, Probability in the Engineering and Informational Sciences.

[42]  A. Y. Mitrophanov,et al.  Sensitivity and convergence of uniformly ergodic Markov chains , 2005 .