On-line sensitivity analysis of Markov chains

Discrete-event systems modeled as continuous-time Markov processes and characterized by some integer-valued parameter are considered. The problem addressed is that of estimating performance sensitivities with respect to this parameter by directly observing a single sample path of the system. The approach is based on transforming the nominal Markov chain into a reduced augmented chain, the stationary-state probabilities which can be easily combined to obtain stationary-state probability sensitivities with respect to the given parameter. Under certain conditions, the reduced augmented chain state transitions are observable with respect to the state transitions of the system itself, and no knowledge of the nominal Markov-chain state of the transition rates is required. Applications for some queueing systems are included. The approach incorporates estimation of unknown transition rates when needed and is extended to real-valued parameters. >

[1]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[2]  Yu-Chi Ho,et al.  A gradient technique for general buffer storage design in a production line , 1979 .

[3]  Christos G. Cassandras,et al.  Infinitesimal and finite perturbation analysis for queueing networks , 1982, 1982 21st IEEE Conference on Decision and Control.

[4]  Christos G. Cassandras,et al.  A new approach to the analysis of discrete event dynamic systems , 1983, Autom..

[5]  Xi-Ren Cao,et al.  Perturbation analysis and optimization of queueing networks , 1983 .

[6]  Xi-Ren Cao.  Convergence of parameter sensitivity estimates in a stochastic experiment , 1984, The 23rd IEEE Conference on Decision and Control.

[7]  Yu-Chi Ho,et al.  Performance sensitivity to routing changes in queuing networks and flexible manufacturing systems using perturbation analysis , 1985, IEEE J. Robotics Autom..

[8]  C. Cassandras.  On-line optimization for a flow control strategy , 1986, 1986 25th IEEE Conference on Decision and Control.

[9]  A. Makowski,et al.  Estimation and optimal control for constrained Markov chains , 1986, 1986 25th IEEE Conference on Decision and Control.

[10]  Alan Weiss,et al.  Sensitivity analysis via likelihood ratios , 1986, WSC '86.

[11]  Peter W. Glynn,et al.  Monte Carlo Optimization of Stochastic Systems: Two New Approaches. , 1986 .

[12]  X. Cao.  Realization probability in closed Jackson queueing networks and its application , 1987, Advances in Applied Probability.

[13]  Y. Ho,et al.  Smoothed (conditional) perturbation analysis of discrete event dynamical systems , 1987 .

[14]  Lee J. Krajewski,et al.  Kanban, MRP, and Shaping the Manufacturing Environment , 1987 .

[15]  Xi-Ren Cao,et al.  Convergence properties of infinitesimal perturbation analysis , 1988 .

[16]  Y. Ho,et al.  Extensions of infinitesimal perturbation analysis , 1988 .

[17]  R. Suri,et al.  Perturbation analysis gives strongly consistent sensitivity estimates for the M/G/ 1 queue , 1988 .