Mixing Time Estimation in Reversible Markov Chains from a Single Sample Path

This article provides the first procedure for computing a fully data-dependent interval that traps the mixing time $t_{\text{mix}}$ of a finite reversible ergodic Markov chain at a prescribed confidence level. The interval is computed from a single finite-length sample path from the Markov chain, and does not require the knowledge of any parameters of the chain. This stands in contrast to previous approaches, which either only provide point estimates, or require a reset mechanism, or additional prior knowledge. The interval is constructed around the relaxation time $t_{\text{relax}}$, which is strongly related to the mixing time, and the width of the interval converges to zero roughly at a $\sqrt{n}$ rate, where $n$ is the length of the sample path. Upper and lower bounds are given on the number of samples required to achieve constant-factor multiplicative accuracy. The lower bounds indicate that, unless further restrictions are placed on the chain, no procedure can achieve this accuracy level before seeing each state at least $\Omega(t_{\text{relax}})$ times on the average. Finally, future directions of research are identified.

[1]  S. Varadhan,et al.  Central limit theorem for additive functionals of reversible Markov processes and applications to simple exclusions , 1986 .

[2]  Don R. Hush,et al.  Learning from dependent observations , 2007, J. Multivar. Anal..

[3]  Yves F. Atchad'e.  Markov chain monte carlo: Confidence intervals , 2016 .

[4]  V. Climenhaga.  Markov chains and mixing times , 2013 .

[5]  D. Freedman.  On Tail Probabilities for Martingales , 1975 .

[6]  Ronitt Rubinfeld,et al.  Testing Closeness of Discrete Distributions , 2010, JACM.

[7]  Joel A. Tropp,et al.  An Introduction to Matrix Concentration Inequalities , 2015, Found. Trends Mach. Learn..

[8]  Gregory Valiant,et al.  Testing Closeness With Unequal Sized Samples , 2015, NIPS.

[9]  Bryan L. Shader,et al.  Applications of Paz's inequality to perturbation bounds for Markov chains , 1998 .

[10]  Benjamin M. Gyori,et al.  Non-asymptotic confidence intervals for MCMC in practice , 2012, 1212.2016.

[11]  Maria-Florina Balcan,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[12]  Richard L. Smith,et al.  Estimating the second largest eigenvalue of a Markov transition matrix , 2000 .

[13]  L. V. D. Heyden,et al.  Perturbation bounds for the stationary probabilities of a finite Markov chain , 1984 .

[14]  Galin L. Jones,et al.  Implementing MCMC: Estimating with Confidence , 2011 .

[15]  S. Bernstein.  Sur l'extension du théoréme limite du calcul des probabilités aux sommes de quantités dépendantes , 1927 .

[16]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[17]  Andreas Christmann,et al.  Fast Learning from Non-i.i.d. Observations , 2009, NIPS.

[18]  Gábor Lugosi,et al.  Introduction to Statistical Learning Theory , 2004, Advanced Lectures on Machine Learning.

[19]  Csaba Szepesvári,et al.  Empirical Bernstein stopping , 2008, ICML '08.

[20]  David Gamarnik.  Extension of the PAC framework to finite and countable Markov chains , 2003, IEEE Trans. Inf. Theory.

[21]  Xiaoji Liu,et al.  On the continuity of the group inverse , 2012 .

[22]  D. Paulin.  Concentration inequalities for Markov chains by Marton couplings and spectral methods , 2012, 1212.2015.

[23]  Mehryar Mohri,et al.  Rademacher Complexity Bounds for Non-I.I.D. Processes , 2008, NIPS.

[24]  Yimin Wei,et al.  An improvement on the perturbation of the group inverse and oblique projection , 2001 .

[25]  Cosma Rohilla Shalizi,et al.  Estimating beta-mixing coefficients , 2011, AISTATS.

[26]  Elchanan Mossel,et al.  The Computational Complexity of Estimating Convergence Time , 2010, ArXiv.

[27]  J. Meyer.  The Role of the Group Generalized Inverse in the Theory of Finite Markov Chains , 1975 .

[28]  E. Seneta.  Sensitivity of finite Markov chains under perturbation , 1993 .

[29]  Mehryar Mohri,et al.  Stability Bounds for Non-i.i.d. Processes , 2007, NIPS.

[30]  Ronitt Rubinfeld,et al.  Testing that distributions are close , 2000, Proceedings 41st Annual Symposium on Foundations of Computer Science.

[31]  C. D. Meyer,et al.  Comparison of perturbation bounds for the stationary distribution of a Markov chain , 2001 .

[32]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[33]  F. Barthe.  Learning from Dependent Observations , 2006 .

[34]  Sean P. Meyn,et al.  Exponential bounds and stopping rules for MCMC and general Markov chains , 2006, valuetools '06.

[35]  David Gamarnik,et al.  Extension of the PAC framework to finite and countable Markov chains , 1999, COLT '99.

[36]  Jun S. Liu,et al.  Monte Carlo strategies in scientific computing , 2001 .

[37]  Elchanan Mossel,et al.  The Computational Complexity of Estimating MCMC Convergence Time , 2011, APPROX-RANDOM.

[38]  Massimiliano Pontil,et al.  Empirical Bernstein Bounds and Sample-Variance Penalization , 2009, COLT.

[39]  Aryeh Kontorovich,et al.  Uniform Chernoff and Dvoretzky-Kiefer-Wolfowitz-Type Inequalities for Markov Chains and Related Processes , 2012, Journal of Applied Probability.

[40]  J Besag,et al.  DISCUSSION ON THE MEETING ON THE GIBBS SAMPLER AND OTHER MARKOV CHAIN-MONTE CARLO METHODS , 1993 .

[41]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[42]  Ravi Montenegro,et al.  Mathematical Aspects of Mixing Times in Markov Chains , 2006, Found. Trends Theor. Comput. Sci..

[43]  Bin Yu.  RATES OF CONVERGENCE FOR EMPIRICAL PROCESSES OF STATIONARY MIXING SEQUENCES , 1994 .

[44]  M. Vidyasagar,et al.  Rates of uniform convergence of empirical means with mixing processes , 2002 .

[45]  Galin L. Jones,et al.  Honest Exploration of Intractable Probability Distributions via Markov Chain Monte Carlo , 2001 .

[46]  Thorsten Joachims,et al.  Counterfactual Risk Minimization: Learning from Logged Bandit Feedback , 2015, ICML.

[47]  Csaba Szepesvári,et al.  Exploration-exploitation tradeoff using variance estimates in multi-armed bandits , 2009, Theor. Comput. Sci..

[48]  Y. Peres,et al.  Estimating the Spectral Gap of a Reversible Markov Chain from a Short Trajectory , 2016, 1612.05330.

[49]  Thomas J. Walsh,et al.  Knows what it knows: a framework for self-aware learning , 2008, ICML '08.