Self-Triggered Markov Decision Processes

In this paper, we study Markov Decision Processes (MDPs) with self-triggered strategies, where the idea of selftriggered control is extended to more generic MDP models. This extension broadens the application of self-triggering policies to a broader range of systems. We study the co-design problems of the control policy and the triggering policy to optimize two pre-specified cost criteria. The first cost criterion is introduced by incorporating a pre-specified update penalty into the traditional MDP cost criteria to reduce the use of communication resources. Under this criteria, a novel dynamic programming (DP) equation called DP equation with optimized lookahead to proposed to solve for the self-triggering policy under this criteria. The second self-triggering policy is to maximize the triggering time while still guaranteeing a prespecified level of sub-optimality. Theoretical underpinnings are established for the computation and implementation of both policies. Through a gridworld numerical example, we illustrate the two policies’ effectiveness in reducing sources consumption and demonstrate the trade-offs between resource consumption and system performance.

[1]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[2]  R. Durrett.  Probability: Theory and Examples , 1993 .

[3]  Arnob Ghosh,et al.  Mean-Field Game Approach to Admission Control of an M/M/∞ Queue with Shared Service Cost , 2016, Dyn. Games Appl..

[4]  Jan M. Maciejowski,et al.  ℓasso MPC: Smart regulation of over-actuated systems , 2012, 2012 American Control Conference (ACC).

[5]  E. Feinberg.  Optimality Conditions for Inventory Control , 2016, 1606.00957.

[6]  Sneha A. Dalvi,et al.  Internet of Things for Smart Cities , 2017 .

[7]  Wei Lu,et al.  Wearable Computing for Internet of Things: A Discriminant Approach for Human Activity Recognition , 2019, IEEE Internet of Things Journal.

[8]  Paulo Tabuada,et al.  An introduction to event-triggered and self-triggered control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[9]  Quanyan Zhu,et al.  A Pursuit-Evasion Differential Game with Strategic Information Acquisition , 2021, ArXiv.

[10]  Xiaofeng Wang,et al.  Self-Triggered Feedback Control Systems With Finite-Gain ${\cal L}_{2}$ Stability , 2009, IEEE Transactions on Automatic Control.

[11]  Sandra Hirche,et al.  On the Optimality of Certainty Equivalence for Event-Triggered Control Systems , 2013, IEEE Transactions on Automatic Control.

[12]  John S. Baras,et al.  Optimal Event-Triggered Control of Nondeterministic Linear Systems , 2020, IEEE Transactions on Automatic Control.

[13]  Paulo Tabuada,et al.  Self-triggered linear quadratic control , 2014, Autom..

[14]  Karl Henrik Johansson,et al.  Robust self-triggered control for time-varying and uncertain constrained systems via reachability analysis , 2019, Autom..

[15]  Paulo Tabuada,et al.  To Sample or not to Sample: Self-Triggered Control for Nonlinear Systems , 2008, IEEE Transactions on Automatic Control.

[16]  Hideaki Ishii,et al.  Self-triggered control with tradeoffs in communication and computation , 2018, Autom..

[17]  Quanyan Zhu,et al.  Continuous-Time Markov Decision Processes with Controlled Observations , 2019, 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[18]  Quanyan Zhu,et al.  Cross-Layer Coordinated Attacks on Cyber-Physical Systems: A LQG Game Framework with Controlled Observations , 2020, ArXiv.

[19]  C. A. Cooper,et al.  An optimal stochastic control problem with observation cost , 1971 .

[20]  M. Sangeetha,et al.  Smart supply chain management using internet of things(IoT) and low power wireless communication systems , 2016, 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET).

[21]  Quanyan Zhu,et al.  Infinite-Horizon Linear-Quadratic-Gaussian Control with Costly Measurements , 2020, ArXiv.