Markov property for a function of a Markov chain: A linear algebra approach

Abstract In this paper, we address whether a (probabilistic) function of a finite homogeneous Markov chain still enjoys a Markov-type property. We propose a complete answer to this question using a linear algebra approach. At the core of our approach is the concept of invariance of a set under a matrix. In that sense, the framework of this paper is related to the so-called “geometric approach” in control theory for linear dynamical systems. This allows us to derive a collection of new results under generic assumptions on the original Markov chain. In particular, we obtain a new criterion for a function of a Markov chain to be a homogeneous Markov chain. We provide a deterministic polynomial-time algorithm for checking this criterion. Moreover, a non-standard notion of observability for a linear system will be used. This allows one to show that the set of all stochastic matrices for which our criterion holds, is nowhere dense in the set of stochastic matrices.

[1]  Robert E. Mahony,et al.  Lumpable hidden Markov models-model reduction and reduced complexity filtering , 2000, IEEE Trans. Autom. Control..

[2]  J. Quadrat,et al.  A unified view of aggregation and coherency in networks and Markov chains , 1984 .

[3]  Anne Benoit,et al.  Aggregation of stochastic automata networks with replicas , 2004 .

[4]  Paul J. Schweitzer,et al.  Aggregation Methods for Large Markov Chains , 1983, Computer Performance and Reliability.

[5]  Frank Ball,et al.  Lumpability and marginalisability for continuous-time Markov chains , 1993 .

[6]  P. Buchholz.  Exact and ordinary lumpability in finite Markov chains , 1994, Journal of Applied Probability.

[7]  M. Grinfeld,et al.  Weak lumpability in the k-SAT problem , 2000, Appl. Math. Lett..

[8]  James Ledoux,et al.  Markovian bounds on functions of finite Markov chains , 2001 .

[9]  Herbert A. Simon,et al.  Aggregation of Variables in Dynamic Systems , 1961 .

[10]  L. Gurvits.  Stability of discrete linear inclusion , 1995 .

[11]  G. Rubino,et al.  A finite characterization of weak lumpable Markov processes. Part I: The discrete time case , 1991 .

[12]  W. Wonham.  Linear Multivariable Control: A Geometric Approach , 1974 .

[13]  J. Ledoux.  A geometric invariant in weak lumpability of finite Markov chains , 1997 .

[14]  D. Sorensen,et al.  A Survey of Model Reduction Methods for Large-Scale Systems , 2000 .

[15]  N. L. Lawrie,et al.  Mathematical Computer Performance and Reliability , 1984 .

[16]  G. Rubino,et al.  A finite characterization of weak lumpable Markov processes. Part II: The continuous time case , 1993 .

[17]  W. Stewart,et al.  Quasi Lumpability, Lower-Bounding Coupling Matrices, and Nearly Completely Decomposable Markov Chains , 1997 .

[18]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[19]  D. Barr,et al.  An Eigenvector Condition for Markov Chain Lumpability , 1977 .

[20]  Shun-ichi Amari,et al.  Identifiability of hidden Markov information sources and their minimum degrees of freedom , 1992, IEEE Trans. Inf. Theory.

[21]  G. Basile,et al.  Controlled and conditioned invariants in linear system theory , 1992 .

[22]  Pamela G. Coxson,et al.  Lumpability and Observability of Linear Systems , 1984 .

[23]  James Ledoux,et al.  Author manuscript, published in "Statistics and Probability Letters 25 (1995) 329-339" On Weak Lumpability of Denumerable Markov Chains , 2013 .

[24]  J. Doob.  Stochastic processes , 1953 .

[25]  Eduardo D. Sontag,et al.  Mathematical Control Theory: Deterministic Finite Dimensional Systems , 1990 .

[26]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[27]  Stephen Gilmore,et al.  An Efficient Algorithm for Aggregating PEPA Models , 2001, IEEE Trans. Software Eng..

[28]  William J. Stewart,et al.  Introduction to the numerical solution of Markov Chains , 1994 .

[29]  M. Rosenblatt,et al.  A MARKOVIAN FUNCTION OF A MARKOV CHAIN , 1958 .

[30]  Weak lumpability and pseudo-stationarity of finite markov chains , 2000 .

[31]  P. Kienker.  Equivalence of aggregated Markov models of ion-channel gating , 1989, Proceedings of the Royal Society of London. B. Biological Sciences.

[32]  John G. Kemeny,et al.  Finite Markov chains , 1960 .

[33]  G. Picci.  Geometric methods in stochastic realization and system identification , 1996 .

[34]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[35]  Gene H. Golub,et al.  Matrix computations , 1983 .

[36]  Peter Lancaster,et al.  The theory of matrices , 1969 .

[37]  Brian D. O. Anderson,et al.  The Realization Problem for Hidden Markov Models , 1999, Math. Control. Signals Syst..

[38]  D. R. Barr,et al.  Technical Note - An Eigenvector Condition for Markov Chain Lumpability , 1977, Oper. Res..

[39]  Peter Spreij,et al.  On the Markov property of a finite hidden Markov chain , 2001 .

[40]  G. Blom,et al.  How many random digits are required until given sequences are obtained? , 1982, Journal of Applied Probability.

[41]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[42]  G. Golub,et al.  A Fast Two-Stage Algorithm for Computing PageRank , 2003 .

[43]  Robert J. Plemmons,et al.  Nonnegative Matrices in the Mathematical Sciences , 1979, Classics in Applied Mathematics.

[44]  J. Ledoux.  Linear dynamics for the state vector of Markov chain functions , 2004, Advances in Applied Probability.