Exponential bounds and stopping rules for MCMC and general Markov chains

We develop explicit, general bounds for the probability that the empirical sample averages of a function of a Markov chain on a general alphabet will exceed the steady-state mean of that function by a given amount. Our bounds combine simple information-theoretic ideas together with techniques from optimization and some fairly elementary tools from analysis. In one direction, motivated by central problems in simulation, we develop bounds for the general class of "geometrically ergodic" Markov chains. These bounds take a form that is particularly suited to simulation problems, and they naturally lead to a new class of sampling criteria. These are illustrated by several examples. In another direction, we obtain a new bound for the important special class of Doeblin chains; this bound is optimal, in the sense that in the special case of independent and identically distributed random variables it essentially reduces to the classical Hoeffding bound.

[1]  J. R. Baxter,et al.  Some familiar examples for which the large deviation principle does not hold , 1991 .

[2]  A. Dembo,et al.  Large deviations and strong mixing , 1996 .

[3]  Peter W. Glynn,et al.  Approximating Martingales for Variance Reduction in Markov Process Simulation , 2002, Math. Oper. Res..

[4]  Leszek Wojnar,et al.  Image Analysis , 1998 .

[5]  Gerhard Winkler,et al.  Image analysis, random fields and dynamic Monte Carlo methods: a mathematical introduction , 1995, Applications of mathematics.

[6]  Sean P. Meyn.  Workload models for stochastic networks: value functions and performance evaluation , 2005, IEEE Transactions on Automatic Control.

[7]  I. Csiszár.  Sanov Property, Generalized $I$-Projection and a Conditional Limit Theorem , 1984 .

[8]  Sean P. Meyn,et al.  A Liapounov bound for solutions of the Poisson equation , 1996 .

[9]  P. Ney,et al.  MARKOV ADDITIVE PROCESSES II. LARGE DEVIATIONS , 1987 .

[10]  Gerhard Winkler,et al.  Image Analysis, Random Fields and Markov Chain Monte Carlo Methods: A Mathematical Introduction , 2002 .

[11]  Charuhas Pandit.  Robust Statistical Modeling Based on Moment Classes, With Applications to Admission Control, Large Deviations and Hypothesis Testing , 2004 .

[12]  Søren Asmussen,et al.  Queueing Simulation in Heavy Traffic , 1992, Math. Oper. Res..

[13]  Sean P. Meyn,et al.  Performance Evaluation and Policy Selection in Multiclass Networks , 2003, Discret. Event Dyn. Syst..

[14]  Imre Csisźar,et al.  The Method of Types , 1998, IEEE Trans. Inf. Theory.

[15]  P. Glynn,et al.  Some New Perspectives on the Method of Control Variates , 2002 .

[16]  P. Glynn,et al.  Hoeffding's inequality for uniformly ergodic Markov chains , 2002 .

[17]  W. Whitt.  Planning queueing simulations , 1989 .

[18]  Peter Ney,et al.  Large deviation lower bounds for arbitrary additive functionals of a Markov chain , 1998 .

[19]  P. Ney,et al.  Large deviations of uniformly recurrent Markov additive processes , 1985 .

[20]  P. Ney,et al.  Markov Additive Processes I. Eigenvalue Properties and Limit Theorems , 1987 .

[21]  Sean P. Meyn,et al.  Variance Reduction for Simulation in Multiclass Queueing Networks , 1999 .

[22]  D. Vere-Jones.  Markov Chains , 1972, Nature.

[23]  S. R. S. Varadhan,et al.  Chapter Nine. Large Deviations and Entropy , 2003 .

[24]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[25]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[26]  Byoung-Seon Choi,et al.  Conditional limit theorems under Markov conditioning , 1987, IEEE Trans. Inf. Theory.

[27]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[28]  Jun S. Liu,et al.  Monte Carlo strategies in scientific computing , 2001 .

[29]  Paul Dupuis,et al.  A nonstandard form of the rate function for the occupation measure of a Markov chain , 1996 .

[30]  E. Çinlar.  Markov additive processes. I , 1972 .

[31]  T. Stephenson.  Image analysis , 1992, Nature.

[32]  Averill M. Law,et al.  Simulation Modeling and Analysis , 1982 .

[33]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[34]  S. Varadhan.  Large Deviations and Applications , 1984 .

[35]  S. Meyn.  Large deviation asymptotics and control variates for simulating large functions , 2006, math/0603328.

[36]  E. Çinlar.  Markov additive processes. II , 1972 .

[37]  R. Ellis,et al.  Entropy, large deviations, and statistical mechanics , 1985 .

[38]  S. Varadhan,et al.  Asymptotic evaluation of certain Markov process expectations for large time , 1975 .

[39]  Mike Chen,et al.  Reliability by design in distributed power transmission networks , 2006, Autom..