Markov Chains and Stochastic Stability

Meyn & Tweedie is back! The bible on Markov chains in general state spaces has been brought up to date to reflect developments in the field since 1996 - many of them sparked by publication of the first edition. The pursuit of more efficient simulation algorithms for complex Markovian models, or algorithms for computation of optimal policies for controlled Markov models, has opened new directions for research on Markov chains. As a result, new applications have emerged across a wide range of topics including optimisation, statistics, and economics. New commentary and an epilogue by Sean Meyn summarise recent developments and references have been fully updated. This second edition reflects the same discipline and style that marked out the original and helped it to become a classic: proofs are rigorous and concise, the range of applications is broad and knowledgeable, and key ideas are accessible to practitioners with limited mathematical background.

[1]  K. Yosida,et al.  Operator-Theoretical Treatment of Markoff's Process and Mean Ergodic Theorem , 1941 .

[2]  P. Moran.  The Statistical Analsis of the Canadian Lynx cycle. 1. Structure and Prediction. , 1953 .

[3]  Walter L. Smith,et al.  Regenerative stochastic processes , 1955, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[4]  M. D. Moustafa,et al.  Input-Output Markov Processes 1) , 1957 .

[5]  Edward Nelson.  The adjoint Markoff process , 1958 .

[6]  S. Orey.  Recurrent Markov chains , 1959 .

[7]  W. L. Smith.  Remarks on the paper ‘Regenerative stochastic processes’ , 1960, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[8]  R. H. Farrell.  Asymptotic renewal theorems in the absolutely continuous case , 1962 .

[9]  D. Vere-Jones.  GEOMETRIC ERGODICITY IN DENUMERABLE MARKOV CHAINS , 1962 .

[10]  George Finlay Simmons,et al.  Introduction to Topology and Modern Analysis , 1963 .

[11]  D. Vere-Jones.  A Rate of Convergence Problem in the Theory of Queues , 1964 .

[12]  M. Rosenblatt.  Equicontinuous Markov Operators , 1964 .

[13]  H. D. Miller.  Geometric ergodicity in a class of denumerable Markov chains , 1966 .

[14]  Z. Šidák.  Classification of Markov chains with a general state space , 1966 .

[15]  C. J. Stone,et al.  On Absolutely Continuous Components and Renewal Theory , 1966 .

[16]  E. Koenigsberg,et al.  Queues and Inventories , 1966 .

[17]  W. Rudin.  Real and complex analysis , 1968 .

[18]  K. Parthasarathy,et al.  Probability measures on metric spaces , 1967 .

[19]  D. Vere-Jones.  Ergodic properties of nonnegative matrices. II , 1967 .

[20]  A. G. Pakes,et al.  Some Conditions for Ergodicity and Recurrence of Markov Chains , 1969, Oper. Res..

[21]  D. Ornstein.  Random walks. II , 1969 .

[22]  D. Ornstein.  Random walks. I , 1969 .

[23]  S. Orey.  Lecture Notes on Limit Theorems for Markov Chain Transition Probabilities , 1971 .

[24]  S. Varadhan,et al.  On degenerate elliptic‐parabolic operators of second order and their associated diffusions , 1972 .

[25]  J. Neveu,et al.  Potentiel markovien récurrent des chaînes de Harris , 1972 .

[26]  S. Varadhan,et al.  On the Support of Diffusion Processes with Applications to the Strong Maximum Principle , 1972 .

[27]  J. Teugels.  An example on geometric ergodicity of a finite Markov chain , 1972, Journal of Applied Probability.

[28]  M. Rosenblatt.  Invariant and subinvariant measures of transition probability functions acting on continuous functions , 1973 .

[29]  M. Rosenblatt.  Recurrent points and transition functions acting on continuous functions , 1974 .

[30]  R. Sine.  Convergence theorems for weakly almost periodic Markov operators , 1974 .

[31]  B. Jamison,et al.  Sample path convergence of stable markov processes , 1974 .

[32]  J. Pitman.  Uniform rates of convergence for Markov chain transition probabilities , 1974 .

[33]  R. Tweedie.  $R$-Theory for Markov Chains on a General State Space II: $r$-Subinvariant Measures for $r$-Transient Chains , 1974 .

[34]  R. Tweedie.  $R$-Theory for Markov Chains on a General State Space I: Solidarity Properties and $R$-Recurrent Chains , 1974 .

[35]  R. Sine.  On local uniform mean convergence for Markov operators. , 1975 .

[36]  R. Tweedie.  Sufficient conditions for ergodicity and recurrence of Markov chains on a general state space , 1975 .

[37]  R. Tweedie,et al.  R -Theory for Markov Chains on a Topological State Space I , 1975 .

[38]  R. Tweedie.  RELATIONS BETWEEN ERGODICITY AND MEAN DRIFT FOR MARKOV CHAINS1 , 1975 .

[39]  J. Neveu,et al.  Discrete Parameter Martingales , 1975 .

[40]  R. Tweedie,et al.  R-theory for Markov chains on a topological state Space. II , 1976 .

[41]  R. Tweedie.  Criteria for classifying general Markov chains , 1976, Advances in Applied Probability.

[42]  J. Snyders.  Stationary Probability Distributions for Linear Time-Invariant Systems , 1977 .

[43]  E. Nummelin.  Uniform and ratio limit theorems for Markov renewal and semi-regenerative processes on a general state space , 1978 .

[44]  M. Neuts.  Markov chains with applications in queueing theory, which have a matrix-geometric invariant probability vector , 1978, Advances in Applied Probability.

[45]  Jean-François Mertens,et al.  Necessary and sufficient conditions for recurrence and transience of Markov chains, in terms of inequalities , 1978 .

[46]  R. Tweedie,et al.  Geometric Ergodicity and R-positivity for General Markov Chains , 1978 .

[47]  R. Tweedie,et al.  The recurrence structure of general Markov processes , 1979, Advances in Applied Probability.

[48]  R. Tweedie,et al.  Markov Chains with Continuous Components , 1979 .

[49]  Z. Rosberg,et al.  A note on the ergodicity of Markov chains , 1981, Journal of Applied Probability.

[50]  S. Saperstone.  Semidynamical Systems in Infinite Dimensional Spaces , 1981 .

[51]  H. Tong.  A note on a Markov bilinear stochastic process in discrete time , 1981 .

[52]  R. Tweedie.  Operator-geometric stationary distributions for markov chains, with application to queueing models , 1982, Advances in Applied Probability.

[53]  E. Nummelin,et al.  Geometric ergodicity of Harris recurrent Marcov chains with applications to renewal theory , 1982 .

[54]  Marcel F. Neuts,et al.  Matrix-geometric solutions in stochastic models - an algorithmic approach , 1982 .

[55]  B. G. Quinn,et al.  Random Coefficient Autoregressive Models: An Introduction , 1982 .

[56]  R. Tweedie.  The existence of moments for stationary Markov chains , 1983, Journal of Applied Probability.

[57]  R. L. Tweedie,et al.  Probability, Statistics and Analysis: Criteria for rates of convergence of Markov chains, with application to queueing and storage theory , 1983 .

[58]  Linn I. Sennott,et al.  Technical Note - Mean Drifts and the Non-Ergodicity of Markov Chains , 1983, Oper. Res..

[59]  E. Nummelin,et al.  The rate of convergence in Orey's theorem for Harris recurrent Markov chains with applications to renewal theory , 1983 .

[60]  J. Petruccelli,et al.  A threshold AR(1) model , 1984, Journal of Applied Probability.

[61]  E. Nummelin.  General irreducible Markov chains and non-negative operators: Preface , 1984 .

[62]  W. Szpankowski.  Some sufficient conditions for non-ergodicity of markov chains , 1985, Journal of Applied Probability.

[63]  B. Ydstie.  Bifurcations and complex dynamics in adaptive control systems , 1986, 1986 25th IEEE Conference on Decision and Control.

[64]  L. Stettner.  On the Existence and Uniqueness of Invariant Measure for Continuous Time Markov Processes , 1986 .

[65]  M. Pourahmadi.  ON STATIONARITY OF THE SOLUTION OF A DOUBLY STOCHASTIC MODEL , 1986 .

[66]  Sean P. Meyn,et al.  A NEW APPROACH TO STOCHASTIC .4DAPTIVE CONTROL , 1986 .

[67]  S. Resnick.  Extreme Values, Regular Variation, and Point Processes , 1987 .

[68]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[69]  Adrian F. M. Smith,et al.  Bayesian computation via the gibbs sampler and related markov chain monte carlo methods (with discus , 1993 .

[70]  R. Tweedie.  Invariant measures for Markov chains with no irreducibility assumptions , 1988 .

[71]  David D. Yao,et al.  Second-Order Properties of the Throughput of a Closed Queueing Network , 1988, Math. Oper. Res..

[72]  Sean P. Meyn,et al.  Stability, convergence, and performance of an adaptive control algorithm applied to a randomly varying system , 1989, Proceedings of the 28th IEEE Conference on Decision and Control,.

[73]  W. Rosenkrantz.  Ergodicity conditions for two-dimensional Markov chains on the positive quadrant , 1989 .

[74]  Sean P. Meyn,et al.  Stochastic controllability and stochastic Lyapunov functions with applications to adaptive and nonlinear systems , 1989 .

[75]  S. Meyn.  Ergodic theorems for discrete time stochastic systems using a stochastic lyapunov function , 1989 .

[76]  D. Pollard,et al.  Simulation and the Asymptotics of Optimization Estimators , 1989 .

[77]  V. Solo.  Stochastic adaptive control and Martingale limit theory , 1990 .

[78]  K. Sigman.  The stability of open queueing networks , 1990 .

[79]  D. Tjøstheim.  Non-linear time series and Markov chains , 1990, Advances in Applied Probability.

[80]  H. Tong.  Non-linear time series. A dynamical system approach , 1990 .

[81]  E. Nummelin.  On the Poisson equation in the potential theory of a single kernel. , 1991 .

[82]  Joanna Mitro.  General theory of markov processes , 1991 .

[83]  Sean P. Meyn,et al.  Asymptotic behavior of stochastic systems possessing Markovian realizations , 1991 .

[84]  S. Meyn,et al.  Stability of Markovian processes I: criteria for discrete-time Chains , 1992, Advances in Applied Probability.

[85]  J. Rosenthal.  Rates of convergence for Gibbs sampler and other Markov chains , 1992 .

[86]  Sean P. Meyn,et al.  Bayesian adaptive control of time varying systems , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[87]  Sean P. Meyn,et al.  Generalized Resolvents and Harris Recurrence of Markov Processes , 1992 .

[88]  Alan E. Gelfand,et al.  Bayesian statistics without tears: A sampling-resampling perspective , 1992 .

[89]  L. Tierney.  Exploring Posterior Distributions Using Markov Chains , 1992 .

[90]  Sean P. Meyn,et al.  Model reference adaptive control of time varying and stochastic systems , 1993, IEEE Trans. Autom. Control..

[91]  S. Meyn,et al.  Stability of Markovian processes III: Foster–Lyapunov criteria for continuous-time processes , 1993, Advances in Applied Probability.

[92]  L. Tang.  Limit theorems for Markov random walks , 1993 .

[93]  S. Meyn,et al.  Stability of Markovian processes II: continuous-time processes and sampled chains , 1993, Advances in Applied Probability.

[94]  Nicholas G. Polson,et al.  On the Geometric Convergence of the Gibbs Sampler , 1994 .

[95]  J. Rosenthal.  Minorization Conditions and Convergence Rates for Markov Chain Monte Carlo , 1995 .

[96]  Adrian F. M. Smith,et al.  Simple conditions for the convergence of the Gibbs sampler and Metropolis-Hastings algorithms , 1994 .

[97]  R. Tweedie,et al.  Strengthening ergodicity to geometric ergodicity for markov chains , 1994 .

[98]  Sean P. Meyn,et al.  Stability of Generalized Jackson Networks , 1994 .

[99]  Sean P. Meyn,et al.  State-Dependent Criteria for Convergence of Markov Chains , 1994 .

[100]  Bin Yu,et al.  Regeneration in Markov chain samplers , 1995 .

[101]  A. Stolyar.  On the Stability of Multiclass Queueing Networks: A Relaxed SuÆcient Condition via Limiting Fluid Processes , .