Singularly Perturbed Markov Chains and Applications to Large-Scale Systems under Uncertainty

This chapter is concerned with large-scale hybrid stochastic systems, in which the dynamics involve both continuously evolving components and discrete events. Corresponding to different discrete states, the dynamic behavior of the underlying system could be markedly different. To reduce the complexity of these systems, singularly perturbed Markov chains are used to characterize the system. Asymptotic expansions of probability vectors and the structural properties of these Markov chains are provided. The ideas of decomposition and aggregation are presented using two typical optimal control problems. Such an approach leads to control policies that are simple to obtain and perform nearly as well as the optimal ones with substantially reduced complexity.

[1]  R. Z. Khasminskii,et al.  Singularly Perturbed Switching Diffusions: Rapid Switchings and Fast Diffusions , 1999 .

[2]  G. Badowski,et al.  Occupation Measures of Singularly Perturbed Markov Chains with Absorbing States , 2000 .

[3]  Qing Zhang,et al.  Discrete-time dynamic systems arising from singularly perturbed Markov chains , 2001 .

[4]  J. Filar,et al.  Algorithms for singularly perturbed limiting average Markov control problems , 1992 .

[5]  Gang George Yin,et al.  Asymptotic Expansions of Singularly Perturbed Systems Involving Rapidly Fluctuating Markov Chains , 1996, SIAM J. Appl. Math..

[6]  George Yin,et al.  ERROR BOUNDS FOR OCCUPATION MEASURE OF SINGULARLY PERTURBED MARKOV CHAINS INCLUDING TRANSIENT STATES , 2000, Probability in the Engineering and Informational Sciences.

[7]  G. Yin,et al.  Asymptotic Expansions of Solutions of Integro- Differential Equations for Transition Densities of Singularly Perturbed Switching Diffusions: Rapid Switchings , 1999 .

[8]  A. A. Pervozvanskiĭ,et al.  Theory of Suboptimal Decisions: Decomposition and Aggregation , 1988 .

[9]  W. A. Thompson,et al.  Point Process Models with Applications to Safety and Reliability , 1988 .

[10]  Qing Zhang,et al.  A central limit theorem for singularly perturbed nonstationary finite state Markov chains , 1996 .

[11]  P. Glynn,et al.  Discrete-time conversion for simulating finite-horizon Markov processes , 1990 .

[12]  H. Kushner.  Weak Convergence Methods and Singularly Perturbed Stochastic Control and Filtering Problems , 1990 .

[13]  Frederick S. Hillier,et al.  Introduction of Operations Research , 1967 .

[14]  Qing Zhang,et al.  Finite state markovian decision processes with weak and strong interactions , 1996 .

[15]  Pravin Varaiya,et al.  Stochastic Systems: Estimation, Identification, and Adaptive Control , 1986 .

[16]  John N. Tsitsiklis,et al.  Statistical Multiplexing of Multiple Time-Scale Markov Streams , 1995, IEEE J. Sel. Areas Commun..

[17]  R. H. Liu.  Nearly optimal control of singularly perturbed Markov decision processes in discrete time , 2001 .

[18]  Marius Iosifescu,et al.  Finite Markov Processes and Their Applications , 1981 .

[19]  Mo Yu,et al.  A first order approximation forthe convergence of distributionsof the cox processes with , 1995 .

[20]  W. Miranker,et al.  Multitime Methods for Systems of Difference Equations , 1977 .

[21]  Qing Zhang,et al.  Structural properties of Markov chains with weak and strong interactions , 1997 .

[22]  Qing Zhang,et al.  Continuous-Time Markov Chains and Applications , 1998 .

[23]  G. Olsder.  New trends in dynamic games and applications , 1995 .

[24]  Pierre L'Ecuyer,et al.  Global Stochastic Optimization with Low-Dispersion Point Sets , 1998, Oper. Res..

[25]  S. Karlin,et al.  A second course in stochastic processes , 1981 .

[26]  Mark H. Davis.  Markov Models and Optimization , 1995 .

[27]  S. Yakowitz,et al.  Machine learning and nonparametric bandit theory , 1995, IEEE Trans. Autom. Control..

[28]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[29]  Qing Zhang,et al.  Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states , 2000 .

[30]  Kai Lai Chung,et al.  Markov Chains with Stationary Transition Probabilities , 1961 .

[31]  S. Yakowitz.  Mathematics of adaptive control processes , 1969 .

[32]  T. Başar,et al.  H∞-Control of Markovian Jump Systems and Solutions to Associated Piecewise-Deterministic Differential Games , 1995 .

[33]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[34]  G. B. Di Masi,et al.  A first order approximation for the convergence of distributions of the cox processes with fast Markov switchings , 1995 .

[35]  George Yin,et al.  On Transition Densities of Singularly Perturbed Diffusions with Fast and Slow Components , 1996, SIAM J. Appl. Math..

[36]  Qing Zhang,et al.  Singularly Perturbed Markov Chains , 2000 .

[37]  K. Åström.  Introduction to Stochastic Control Theory , 1970 .

[38]  David A. Kendrick,et al.  On the Leontief Dynamic Inverse , 1972 .

[39]  Gang George Yin,et al.  Decomposition and aggregation of large-dimensional Markov chains in discrete time , 2001, Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228).

[40]  Samuel Karlin,et al.  A First Course on Stochastic Processes , 1968 .

[41]  Stanley B. Gershwin,et al.  Manufacturing Systems Engineering , 1993 .

[42]  Gang George Yin,et al.  On nearly optimal controls of hybrid LQG problems , 1999, IEEE Trans. Autom. Control..

[43]  Qing Zhang,et al.  Controlled Markov Chains with Weak and Strong Interactions: Asymptotic Optimality and Applications to Manufacturing , 1997 .

[44]  George Yin,et al.  Singularly Perturbed Multidimensional Switching Diffusions with Fast and Slow Switchings , 1999 .

[45]  François Delebecque,et al.  Optimal control of markov chains admitting strong and weak interactions , 1981, Autom..

[46]  Sidney J. Yakowitz,et al.  Automatic Learning for Dynamic Markov Fields with Application to Epidemiology , 1992, Oper. Res..

[47]  Qing Zhang,et al.  Hierarchical Decision Making in Stochastic Manufacturing Systems , 1994 .

[48]  Gang George Yin,et al.  Singularly Perturbed Discrete-Time Markov Chains , 2000, SIAM J. Appl. Math..

[49]  George Yin.  On Limit Results for a Class of Singularly Perturbed Switching Diffusions , 2001 .

[50]  Valerie Isham,et al.  Non‐Negative Matrices and Markov Chains , 1983 .

[51]  Herbert A. Simon,et al.  Aggregation of Variables in Dynamic Systems , 1961 .

[52]  R. Z. Khasminskii,et al.  Constructing asymptotic series for probability distributions of Markov chains with weak and strong interactions , 1997 .

[53]  Stephen P. Brooks,et al.  Markov Decision Processes. , 1995 .

[54]  James D. Hamilton,et al.  Autoregressive conditional heteroskedasticity and changes in regime , 1994 .

[55]  V. G. Gaitsgori,et al.  Theory of Suboptimal Decisions , 1988 .

[56]  Harold J. Kushner,et al.  Stochastic Approximation Algorithms and Applications , 1997, Applications of Mathematics.

[57]  W. Fleming,et al.  Deterministic and Stochastic Optimal Control , 1975 .

[58]  G. Blankenship.  Singularly perturbed difference equations in optimal control problems , 1981 .

[59]  P. Billingsley,et al.  Convergence of Probability Measures , 1970, The Mathematical Gazette.

[60]  Qing Zhang,et al.  Control of Dynamic Systems under the Influence of Singularly Perturbed Markov Chains , 1997 .

[61]  Qing Zhang.  Risk-Sensitive Production Planning of Stochastic Manufacturing Systems: A Singular Perturbation Approach , 1995 .

[62]  P. Kokotovic,et al.  A singular perturbation approach to modeling and control of Markov chains , 1981 .

[63]  G. Yin,et al.  Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach , 1997 .

[64]  Qing Zhang,et al.  Control of singularly perturbed Markov chains: A numerical study , 2003, The ANZIAM Journal.