Augmented truncation approximations to the solution of Poisson's equation for Markov chains

Poisson’s equation has a lot of applications in various areas. Usually it is hard to derive the explicit expression of the solution of Poisson’s equation for a Markov chain on an infinitely many state space. We will present a computational framework for the solution for both discretetime Markov chains (DTMCs) and continuous-time Markov chains (CTMCs), by developing the technique of augmented truncation approximations. The convergence to the solution is investigated in terms of the assumption about the monotonicity of the first return times, and is further established for two types of truncation approximation schemes: the censored chain and the linear augmented truncation. Moreover, truncation approximations to the variance constant in central limit theorems (CLTs) are also considered. The results obtained are applied to discrete-time single-birth processes and continuous-time single-death processes.

[1]  Yuanyuan Liu,et al.  Hoeffding’s inequality for Markov processes via solution of Poisson’s equation , 2021, Frontiers of Mathematics in China.

[2]  Upendra Dave,et al.  Applied Probability and Queues , 1987 .

[3]  Hiroyuki Masuyama.  Error Bounds for Augmented Truncations of Discrete-Time Block-Monotone Markov Chains under Subgeometric Drift Conditions , 2016, SIAM J. Matrix Anal. Appl..

[4]  Eugene Seneta,et al.  Augmented truncations of infinite stochastic matrices , 1987 .

[5]  Galin L. Jones.  On the Markov chain central limit theorem , 2004, math/0409112.

[6]  Yu-hui Zhang.  Criteria on ergodicity and strong ergodicity of single death processes , 2018, Frontiers of Mathematics in China.

[7]  P. Glynn,et al.  Hoeffding's inequality for uniformly ergodic Markov chains , 2002 .

[8]  Sean P. Meyn,et al.  A Liapounov bound for solutions of the Poisson equation , 1996 .

[9]  Yiqiang Q. Zhao,et al.  Subgeometric ergodicity for continuous-time Markov chains , 2010 .

[10]  Yiqiang Q. Zhao,et al.  Censoring technique and numerical computations of invariant distribution for continuous-time Markov chains , 2015 .

[11]  Mu-Fa Chen,et al.  Unified representation of formulas for single birth processes , 2014, 1411.6099.

[12]  Tom Burr,et al.  Introduction to Matrix Analytic Methods in Stochastic Modeling , 2001, Technometrics.

[13]  Yuanyuan Liu.  Perturbation analysis for continuous-time Markov chains , 2015 .

[14]  Hiroyuki Masuyama,et al.  Error Bounds for Augmented Truncations of Discrete-Time Block-Monotone Markov Chains under Geometric Drift Conditions , 2015, Advances in Applied Probability.

[15]  Yuanyuan Liu,et al.  Augmented truncation approximations of discrete-time Markov chains , 2010, Oper. Res. Lett..

[16]  Eugene Seneta,et al.  Computing the stationary distribution for infinite Markov chains , 1980 .

[17]  Dimitri P. Bertsekas,et al.  Dynamic programming and optimal control, 3rd Edition , 2005 .

[18]  Jing Wang,et al.  Moments of integral-type downward functionals for single death processes , 2020, Frontiers of Mathematics in China.

[19]  P. Glynn,et al.  Necessary conditions in limit theorems for cumulative processes , 2002 .

[20]  Kai Lai Chung,et al.  Markov Chains with Stationary Transition Probabilities , 1961 .

[21]  Danielle Liu,et al.  The censored Markov chain and the best augmentation , 1996, Journal of Applied Probability.

[22]  Michael C. H. Choi,et al.  A Hoeffding’s inequality for uniformly ergodic diffusion process , 2019, Statistics & Probability Letters.

[23]  J. Rosenthal,et al.  General state space Markov chains and MCMC algorithms , 2004, math/0404033.

[24]  A. Shwartz,et al.  The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations , 2002 .

[25]  Yuanyuan Liu,et al.  Central limit theorems for ergodic continuous-time Markov chains with applications to single birth processes , 2015 .

[26]  Yuanyuan Liu,et al.  Poisson’s equation for discrete-time single-birth processes , 2014 .

[27]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[28]  Mogens Bladt,et al.  Poisson's equation for queues driven by a Markovian marked point process , 1994, Queueing Syst. Theory Appl..

[29]  L. Yuan.  Censoring technique and numerical computations of invariant distribution for continuous-time Markov chains , 2015 .

[30]  Yuanyuan Liu,et al.  Error bounds for augmented truncation approximations of Markov chains via the perturbation method , 2018, Advances in Applied Probability.