Singularly Perturbed Discrete-Time Markov Chains

Originating from a wide range of applications in optimization and control of large-scale systems (such as telecommunications, queueing networks, and manufacturing systems), this work is devoted to a class of singularly perturbed discrete-time Markov chains. The states of the Markov chain are naturally decomposable into recurrent and transient classes such that within each class the interactions are strong and among different classes the interactions are weak. Our study focuses on the difference equations representing the probabilityvector, and aims at deriving matched asymptotic expansions of the solutions. Justification and error analysis are also provided. Both time-homogeneous and time-inhomogeneous models are examined.

[1]  P. Zweifel.  Advanced Mathematical Methods for Scientists and Engineers , 1980 .

[2]  T. Başar,et al.  H∞-Control of Markovian Jump Systems and Solutions to Associated Piecewise-Deterministic Differential Games , 1995 .

[3]  A. A. Pervozvanskiĭ,et al.  Theory of Suboptimal Decisions: Decomposition and Aggregation , 1988 .

[4]  W. Fleming,et al.  Deterministic and Stochastic Optimal Control , 1975 .

[5]  François Delebecque,et al.  Optimal control of markov chains admitting strong and weak interactions , 1981, Autom..

[6]  D. Naidu.  Singular Perturbation Methodology in Control Systems , 1988 .

[7]  J. Cole,et al.  Multiple Scale and Singular Perturbation Methods , 1996 .

[8]  Harold J. Kushner,et al.  Stochastic Approximation Algorithms and Applications , 1997, Applications of Mathematics.

[9]  W. Miranker,et al.  Multitime Methods for Systems of Difference Equations , 1977 .

[10]  D. Cooke,et al.  Finite Markov Processes and Their Applications , 1981 .

[11]  R. H. Liu.  Nearly optimal control of singularly perturbed Markov decision processes in discrete time , 2001 .

[12]  Cyrus Derman,et al.  Finite State Markovian Decision Processes , 1970 .

[13]  J. Filar,et al.  Algorithms for singularly perturbed limiting average Markov control problems , 1990, 29th IEEE Conference on Decision and Control.

[14]  Herbert A. Simon,et al.  Aggregation of Variables in Dynamic Systems , 1961 .

[15]  Andrew L. Rukhin,et al.  Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach , 2001, Technometrics.

[16]  R. Z. Khasminskii,et al.  Constructing asymptotic series for probability distributions of Markov chains with weak and strong interactions , 1997 .

[17]  G. Yin,et al.  Continuous-Time Markov Chains and Applications: A Singular Perturbation Approach , 1997 .

[18]  Stephen P. Brooks,et al.  Markov Decision Processes. , 1995 .

[19]  Gang George Yin,et al.  Asymptotic Expansions of Singularly Perturbed Systems Involving Rapidly Fluctuating Markov Chains , 1996, SIAM J. Appl. Math..

[20]  G. Blankenship.  Singularly perturbed difference equations in optimal control problems , 1981 .

[21]  John N. Tsitsiklis,et al.  Statistical Multiplexing of Multiple Time-Scale Markov Streams , 1995, IEEE J. Sel. Areas Commun..