On Rates of Convergence for Markov Chains Under Random Time State Dependent Drift Criteria

Many applications in networked control require intermittent access of a controller to a system, as in event-triggered systems or information constrained control applications. Motivated by such applications and extending previous work on Lyapunov-theoretic drift criteria, we establish both subgeometric and geometric rates of convergence for Markov chains under state dependent random time drift criteria. We quantify how the rate of ergodicity, nature of Lyapunov functions, their drift properties, and the distributions of stopping times are related. We finally study an application in networked control.

[1]  Karl Henrik Johansson,et al.  Optimal stopping for event-triggered sensing and actuation , 2008, 2008 47th IEEE Conference on Decision and Control.

[2]  Csaba Szepesvári,et al.  Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path , 2006, Machine Learning.

[3]  A. Sapozhnikov.  Subgeometric rates of convergence of f-ergodic Markov chains , 2006 .

[4]  Sean P. Meyn,et al.  State-Dependent Criteria for Convergence of Markov Chains , 1994 .

[5]  Wei Wu,et al.  Optimal Sensor Querying: General Markovian and LQG Models With Controlled Observations , 2008, IEEE Transactions on Automatic Control.

[6]  J. Dai.  On Positive Harris Recurrence of Multiclass Queueing Networks: A Unified Approach Via Fluid Limit Models , 1995 .

[7]  Tamás Linder,et al.  Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Stochastic Control , 2015, IEEE Transactions on Automatic Control.

[8]  I. I. Gikhman.  Convergence to Markov processes , 1969 .

[9]  J. Rosenthal,et al.  General state space Markov chains and MCMC algorithms , 2004, math/0404033.

[10]  Vivek S. Borkar,et al.  Convex Analytic Methods in Markov Decision Processes , 2002 .

[11]  Paulo Tabuada,et al.  Event-Triggered Real-Time Scheduling of Stabilizing Control Tasks , 2007, IEEE Transactions on Automatic Control.

[12]  Michael D. Lemmon,et al.  Event-Triggered Feedback in Control, Estimation, and Optimization , 2010 .

[13]  John C. Kieffer,et al.  Stochastic stability for feedback quantization schemes , 1982, IEEE Trans. Inf. Theory.

[14]  Serdar Yüksel,et al.  Stochastic Stabilization of Noisy Linear Systems With Fixed-Rate Limited Feedback , 2010, IEEE Transactions on Automatic Control.

[15]  R. Zurkowski.  Lyapunov Analysis for Rates of Convergence in Markov Chains and Random-Time State-Dependent Drift , 2013 .

[16]  Nathan van de Wouw,et al.  Stability of Networked Control Systems With Uncertain Time-Varying Delays , 2009, IEEE Transactions on Automatic Control.

[17]  Sean P. Meyn,et al.  Random-Time, State-Dependent Stochastic Drift for Markov Chains and Application to Stochastic Stabilization Over Erasure Channels , 2010, IEEE Transactions on Automatic Control.

[18]  K. Åström,et al.  Comparison of Riemann and Lebesgue sampling for first order stochastic systems , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[19]  Sean P. Meyn.  Control Techniques for Complex Networks: Workload , 2007 .

[20]  E. Nummelin,et al.  A splitting technique for Harris recurrent Markov chains , 1978 .

[21]  Sean P. Meyn,et al.  The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning , 2000, SIAM J. Control. Optim..

[22]  Gersende Fort,et al.  State-dependent Foster–Lyapunov criteria for subgeometric convergence of Markov chains , 2009, 0901.2453.

[23]  Daniel E. Quevedo,et al.  Stochastic Stability of Event-Triggered Anytime Control , 2013, IEEE Transactions on Automatic Control.

[24]  R. Douc,et al.  Practical drift conditions for subgeometric rates of convergence , 2004, math/0407122.

[25]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[26]  Brian H. Fralix.  Foster-type criteria for Markov chains on general spaces , 2006, Journal of Applied Probability.

[27]  Tamer Basar,et al.  Stochastic Networked Control Systems , 2013 .

[28]  Gersende Fort,et al.  ODE methods for skip-free Markov chain stability with applications to MCMC , 2006 .

[29]  Robin J. Evans,et al.  Stabilizability of Stochastic Linear Systems with Finite Feedback Data Rates , 2004, SIAM J. Control. Optim..

[30]  Tamer Basar,et al.  Stochastic Networked Control Systems: Stabilization and Optimization under Information Constraints , 2013 .

[31]  Graham C. Goodwin,et al.  Control over unreliable networks affected by packet erasures and variable transmission delays , 2008, IEEE Journal on Selected Areas in Communications.

[32]  M. K. Ghosh,et al.  Discrete-time controlled Markov processes with average cost criterion: a survey , 1993 .

[33]  Daniel Liberzon,et al.  Quantized feedback stabilization of linear systems , 2000, IEEE Trans. Autom. Control..

[34]  Nuno C. Martins,et al.  Remote State Estimation With Communication Costs for First-Order LTI Systems , 2011, IEEE Transactions on Automatic Control.

[35]  Sean P. Meyn,et al.  Stability and convergence of moments for multiclass queueing networks via fluid limit models , 1995, IEEE Trans. Autom. Control..

[36]  John C. Kieffer,et al.  On a type of stochastic stability for a class of encoding schemes , 1983, IEEE Trans. Inf. Theory.

[37]  K. Athreya,et al.  A New Approach to the Limit Theory of Recurrent Markov Chains , 1978 .

[38]  Dragan Nesic,et al.  Input-to-State Stabilization of Linear Systems With Quantized State Measurements , 2007, IEEE Transactions on Automatic Control.

[39]  Tamer Basar,et al.  Optimal control with limited controls , 2006, 2006 American Control Conference.

[40]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[41]  R. Tweedie,et al.  Subgeometric Rates of Convergence of f-Ergodic Markov Chains , 1994, Advances in Applied Probability.

[42]  Gersende Fort,et al.  The ODE method for stability of skip-free Markov chains with applications to MCMC , 2006, math/0607800.