Likelihood ratio gradient estimation for stochastic recursions

In this paper, we develop mathematical machinery for verifying that a broad class of general state space Markov chains reacts smoothly to certain types of perturbations in the underlying transition structure. Our main result provides conditions under which the stationary probability measure of an ergodic Harris-recurrent Markov chain is differentiable in a certain strong sense. The approach is based on likelihood ratio 'change-of-measure' arguments, and leads directly to a 'likelihood ratio gradient estimator' that can be computed numerically.

[1]  Kai Lai Chung,et al.  A Course in Probability Theory , 1949 .

[2]  Dénes König,et al.  Verallgemeinerungen der Erlangschen und Engsetschen Formeln : Eine Methode in der Bedienungstheorie , 1967 .

[3]  P. Billingsley,et al.  Convergence of Probability Measures , 1970, The Mathematical Gazette.

[4]  P. Schweitzer.  Perturbation theory and finite Markov chains , 1968 .

[5]  D. P. Kennedy.  THE CONTINUITY OF THE SINGLE SERVER QUEUE , 1972 .

[6]  W. Whitt.  The continuity of queues , 1974, Advances in Applied Probability.

[7]  E. Nummelin,et al.  A splitting technique for Harris recurrent Markov chains , 1978 .

[8]  K. Athreya,et al.  A New Approach to the Limit Theory of Recurrent Markov Chains , 1978 .

[9]  Ward Whitt,et al.  Continuity of Generalized Semi-Markov Processes , 1980, Math. Oper. Res..

[10]  R. Tweedie.  The existence of moments for stationary Markov chains , 1983, Journal of Applied Probability.

[11]  E. Nummelin.  General irreducible Markov chains and non-negative operators: Embedded renewal processes , 1984 .

[12]  C. D. Meyer,et al.  Using the QR factorization and group inversion to compute, differentiate ,and estimate the sensitivity of stationary probabilities for markov chains , 1986 .

[13]  Peter W. Glynn,et al.  Optimization of stochastic systems , 1986, WSC '86.

[14]  P. Glynn,et al.  Discrete-time conversion for simulating semi-Markov processes , 1986 .

[15]  Upendra Dave,et al.  Applied Probability and Queues , 1987 .

[16]  P. Glynn.  A GSMP formalism for discrete event systems , 1989, Proc. IEEE.

[17]  Paul Glasserman,et al.  Gradient Estimation Via Perturbation Analysis , 1990 .

[18]  Peter W. Glynn,et al.  Gradient estimation for ratios , 1991, 1991 Winter Simulation Conference Proceedings..

[19]  H. Kushner,et al.  Estimation of the derivative of a stationary measure with respect to a control parameter , 1992 .

[20]  P. Glynn.  Note-Pathwise Convexity and its Relation to Convergence of Time-Average Derivatives , 1992 .

[21]  P. Glynn,et al.  Stochastic Optimization by Simulation: Convergence Proofs for the GI/G/1 Queue in Steady-State , 1994 .

[22]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[23]  Charles Leake,et al.  Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method , 1994 .

[24]  Jason H. Goodfriend,et al.  Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method , 1995 .

[25]  Likelihood ratio gradient estimation for stochastic recursions , 1995, Advances in Applied Probability.