The State Reduction and Related Algorithms and Their Applications to the Study of Markov Chains, Graph Theory, and the Optimal Stopping Problem

Abstract We discuss the State Reduction/GTH (Grassmann, Taksar, Heyman) algorithm for recursively finding invariant measure. We demonstrate the relationship between this algorithm and the Freidlin–Wentzell “tree decomposition” approach to study the characteristics of Markov chains. The structure of the State Reduction/GTH algorithm suggests the natural idea for finding the distribution of a Markov chain at the moment of first visit to a given set, and some similar characteristics. We study the possible range of such algorithms. We also present a new algorithm for solving the classical problem of optimal stopping of a Markov chain based on a similar idea of sequential elimination of some states. We give shorter and more transparent proofs of some previously known results, and improve the bounds of Freidlin–Wentzell in the perturbation theory of Markov chains. Some applications to graph theory are also discussed.

[1]  Theodore J. Sheskin.  Computing the Fundamental Matrix for a Reducible Markov Chain , 1995 .

[2]  John G. Kemeny,et al.  Finite Markov Chains. , 1960 .

[3]  David Siegmund,et al.  Great expectations: The theory of optimal stopping , 1971 .

[4]  P. Diaconis,et al.  Geometric Bounds for Eigenvalues of Markov Chains , 1991 .

[5]  Jürg Kohlas.  Numerical computation of mean passage times and absorption probabilities in Markov and Semi-Markov models , 1986, Z. Oper. Research.

[6]  Dean Isaacson,et al.  Markov Chains: Theory and Applications , 1976 .

[7]  J. Kemeny,et al.  Denumerable Markov chains , 1969 .

[8]  Winfried K. Grassmann,et al.  Equilibrium distribution of block-structured Markov chains with repeating rows , 1990, Journal of Applied Probability.

[9]  Winfried K. Grassmann,et al.  Regenerative Analysis and Steady State Distributions for Markov Chains , 1985, Oper. Res..

[10]  Daniel P. Heyman,et al.  Further comparisons of direct methods for computing stationary distributions of Markov chains , 1987 .

[11]  Kenneth P. Bogart,et al.  Introductory Combinatorics , 1977 .

[12]  U. Narayan Bhat,et al.  Reduced System Algorithms for Markov Chains , 1988 .

[13]  C. O'Cinneide.  Entrywise perturbation theory and error analysis for Markov chains , 1993 .

[15]  Daniel P. Heyman,et al.  Accurate Computation of the Fundamental Matrix of a Markov Chain , 1995, SIAM J. Matrix Anal. Appl..

[16]  Alʹbert Nikolaevich Shiri︠a︡ev,et al.  Optimal stopping rules , 1977 .

[17]  Daniel P. Heyman,et al.  Numerical Solution of Linear Equations Arising in Markov Chain Models , 1989, INFORMS J. Comput..