Perturbation realization, potentials, and sensitivity analysis of Markov processes

Two fundamental concepts and quantities, realization factors and performance potentials, are introduced for Markov processes. The relations among these two quantities and the group inverse of the infinitesimal generator are studied. It is shown that the sensitivity of the steady-state performance with respect to the change of the infinitesimal generator can be easily calculated by using either of these three quantities and that these quantities can be estimated by analyzing a single sample path of a Markov process. Based on these results, algorithms for estimating performance sensitivities on a single sample path of a Markov process can be proposed. The potentials in this paper are defined through realization factors and are shown to be the same as those defined by Poisson equations. The results provide a uniform framework of perturbation realization for infinitesimal perturbation analysis (IPA) and non-IPA approaches to the sensitivity analysis of steady-state performance; they also provide a theoretical background for the PA algorithms developed in recent years.

[1]  A. Jensen Markoff chains as an aid in the study of Markoff processes , 1953 .

[2]  John G. Kemeny,et al.  Finite Markov Chains. , 1960 .

[3]  J. Kemeny,et al.  Potentials for denumerable Markov chains , 1961 .

[4]  J. Kemeny,et al.  Potentials for denumerable Markov chains , 1961 .

[5]  P. Schweitzer Perturbation theory and finite Markov chains , 1968 .

[6]  J. Meyer The Role of the Group Generalized Inverse in the Theory of Finite Markov Chains , 1975 .

[7]  Antoine-S Bailly,et al.  Science régionale - Walter Isard, Introduction to régional science. Englewood Cliffs (NJ), Prentice-Hall, 1975 , 1975 .

[8]  Christos G. Cassandras,et al.  Infinitesimal and finite perturbation analysis for queueing networks , 1983, 1982 21st IEEE Conference on Decision and Control.

[9]  D. Robinson A Course in the Theory of Groups , 1982 .

[10]  Valerie Isham,et al.  Non‐Negative Matrices and Markov Chains , 1983 .

[11]  Convergence of parameter sensitivity estimates in a stochastic experiment , 1984, The 23rd IEEE Conference on Decision and Control.

[12]  Y. Ho,et al.  Smoothed (conditional) perturbation analysis of discrete event dynamical systems , 1987 .

[13]  S. Rolewicz Functional Analysis and Control Theory , 1987 .

[14]  Xi-Ren Cao,et al.  First-Order Perturbation Analysis of a Simple Multi-Class Finite Source Queue , 1987, Perform. Evaluation.

[15]  Xi-Ren Cao,et al.  Convergence properties of infinitesimal perturbation analysis , 1988 .

[16]  Y. Ho,et al.  Extensions of infinitesimal perturbation analysis , 1988 .

[17]  R. Suri,et al.  Perturbation analysis gives strongly consistent sensitivity estimates for the M/G/ 1 queue , 1988 .

[18]  C. Cassandras,et al.  On-line sensitivity analysis of Markov chains , 1989 .

[19]  P. Glynn A Lyapunov Bound for Solutions of Poisson's Equation , 1989 .

[20]  R. Suri,et al.  Perturbation analysis: the state of the art and research issues explained via the GI/G/1 queue , 1989 .

[21]  P. Glasserman The limiting value of derivative estimators based on perturbation analysis , 1990 .

[22]  N. Dijk On a simple proof of uniformization for continuous and discrete-state continuous-time Markov chains , 1990 .

[23]  Paul Glasserman,et al.  Gradient Estimation Via Perturbation Analysis , 1990 .

[24]  Xi-Ren Cao,et al.  Perturbation analysis of discrete event dynamic systems , 1991 .

[25]  Pirooz Vakili,et al.  Using a standard clock technique for efficient simulation , 1991, Oper. Res. Lett..

[26]  George Liberopoulos,et al.  Perturbation Analysis for the Design of Flexible Manufacturing System Flow Controllers , 1992, Oper. Res..

[27]  Sean P. Meyn,et al.  Generalized Resolvents and Harris Recurrence of Markov Processes , 1992 .

[28]  Ramavarapu S. Sreenivas,et al.  Augmented infinitesimal perturbation analysis: An alternate explanation , 1992, Discret. Event Dyn. Syst..

[29]  Pierre Brémaud,et al.  On the pathwise computation of derivatives with respect to the rate of a point process: The phantom RPA method , 1992, Queueing Syst. Theory Appl..

[30]  Pierre Brémaud,et al.  Maximal coupling and rare perturbation sensitivity analysis , 1992, Queueing Syst. Theory Appl..

[31]  M. Fu,et al.  Addendum to "Extensions and generalizations of smoothed perturbation analysis in a generalized semi-Markov process framework" , 1992 .

[32]  S. Meyn,et al.  Stability of Markovian processes III: Foster–Lyapunov criteria for continuous-time processes , 1993, Advances in Applied Probability.

[33]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[34]  E. Chong,et al.  Optimization of queues using an infinitesimal perturbation analysis-based stochastic algorithm with general update times , 1993 .

[35]  S. Meyn,et al.  Stability of Markovian processes II: continuous-time processes and sampled chains , 1993, Advances in Applied Probability.

[36]  Robert J. Plemmons,et al.  Nonnegative Matrices in the Mathematical Sciences , 1979, Classics in Applied Mathematics.

[37]  Michael C. Fu,et al.  Smoothed perturbation analysis derivative estimation for Markov chains , 1994, Oper. Res. Lett..

[38]  Houmin Yan,et al.  Finding optimal number of Kanbans in a manufacturing system via stochastic approximation and perturbation analysis , 1994 .

[39]  A. Shwartz,et al.  On the Poisson Equation for Countable Markov Chains: Existence of Solutions and Parameter Dependence by Probabilistic Methods , 1994 .

[40]  Edwin K. P. Chong,et al.  Discrete event systems: Modeling and performance analysis , 1994, Discret. Event Dyn. Syst..

[41]  Xi-Ren Cao,et al.  Realization Probabilities: The Dynamics of Queuing Systems , 1994 .

[42]  Christian van Delft,et al.  Convergence of stochastic approximation coupled with perturbation analysis in a class of manufacturing flow control models , 1994, Discret. Event Dyn. Syst..

[43]  L. Dai A consistent algorithm for derivative estimation of Markov chains , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[44]  Felix Wu,et al.  Dynamic bandwidth allocation using infinitesimal perturbation analysis , 1994, Proceedings of INFOCOM '94 Conference on Computer Communications.

[45]  Han-Fu Chen,et al.  Convergence of perturbation analysis based optimization algorithm with fixed number of customers period , 1994, Discret. Event Dyn. Syst..

[46]  Y. Ho,et al.  Structural infinitesimal perturbation analysis (SIPA) for derivative estimation of discrete-event dynamic systems , 1995, IEEE Trans. Autom. Control..

[47]  Cao Xi-ren Uniformization and performance sensitivity estimation in closed queueing networks , 1996 .

[48]  O. Hernández-Lerma,et al.  Existence of solutions to the Poisson equation in L/sub p/ spaces , 1996, Proceedings of 35th IEEE Conference on Decision and Control.

[49]  Xi-Ren Cao,et al.  A single sample path-based performance sensitivity formula for Markov chains , 1996, IEEE Trans. Autom. Control..

[50]  Sean P. Meyn,et al.  A Liapounov bound for solutions of the Poisson equation , 1996 .

[51]  Xi-Ren Cao,et al.  Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization , 1998, IEEE Trans. Control. Syst. Technol..