Numerical methods for controlled and uncontrolled multiplexing and queueing systems

We deal with a very useful numerical method for both controlled and uncontrolled queuing and multiplexing type systems. The basic idea starts with a heavy traffic approximation, but it is shown that the results are very good even when working far from the heavy traffic regime. The underlying numerical method is a version of what is known as the Markov chain approximation method. It is a powerful methodology for controlled and uncontrolled stochastic systems, which can be approximated by diffusion or reflected diffusion type systems, and has been used with success on many other problems in stochastic control. We give a complete development of the relevant details, with an emphasis on multiplexing and particular queueing systems. The approximating process is a controlled or uncontrolled Markov chain which retains certain essential features of the original problem. This problem is generally substantially simpler than the original physical problem, and there are associated convergence theorems. The non-classical associated ergodic cost problem is derived, and put into a form such that reliable and good numerical algorithms, based on multigrid type ideas, can be used. Data for both controlled and uncontrolled problems shows the value of the method.

[1]  Kai Lai Chung,et al.  Markov Chains with Stationary Transition Probabilities , 1961 .

[2]  D. White.  Dynamic programming, Markov chains, and the method of successive approximations , 1963 .

[3]  D. J. White,et al.  Dynamic Programming , 2018, Wiley Encyclopedia of Computer Science and Engineering.

[4]  D. Iglehart,et al.  Multiple channel queues in heavy traffic. I , 1970, Advances in Applied Probability.

[5]  D. Mitra,et al.  Stochastic theory of a data-handling system with multiple sources , 1982, The Bell System Technical Journal.

[6]  H. Kushner.  Robustness and Approximation of Escape Times and Large Deviations Estimates for Systems with Small Noise Effects , 1984 .

[7]  Michael I. Taksar,et al.  Average Optimal Singular Control and a Related Stopping Problem , 1985, Math. Oper. Res..

[8]  S. Shreve,et al.  Absolutely continuous and singular stochastic control , 1986 .

[9]  Dimitri P. Bertsekas,et al.  Dynamic Programming: Deterministic and Stochastic Models , 1987 .

[10]  M. Akian.  Re´solution nume´rique d'equations d'Hamilton-Jacobi-Bellman-au moyen d'algorithmes multigrilles et d'ite´rations sur les politiques , 1988 .

[11]  D. Mitra.  Stochastic theory of a fluid model of producers and consumers coupled by a buffer , 1988, Advances in Applied Probability.

[12]  M. Reiman.  Asymptotically optimal trunk reservation for large trunk groups , 1989, Proceedings of the 28th IEEE Conference on Decision and Control,.

[13]  H. Kushner,et al.  Optimal and approximately optimal control policies for queues in heavy traffic , 1989 .

[14]  D. Bertsekas,et al.  Adaptive aggregation methods for infinite horizon dynamic programming , 1989 .

[15]  Lawrence M. Wein,et al.  Optimal Control of a Two-Station Brownian Network , 2015, Math. Oper. Res..

[16]  Marianne Akian.  Méthodes multigrilles en contrôle stochastique , 1990 .

[17]  J. Michael Harrison,et al.  The QNET method for two-moment analysis of open queueing networks , 1990, Queueing Syst. Theory Appl..

[18]  L. F. Martins,et al.  Routing and singular control for queueing networks in heavy traffic , 1990 .

[19]  C. Knessl,et al.  Heavy-traffic analysis of a data-handling system with many sources , 1991 .

[20]  L. F. Martins,et al.  Numerical Methods for Stochastic Singular Control Problems , 1991 .

[21]  J. Dai.  Steady-state analysis of reflected Brownian motions: characterization, numerical methods and queueing applications , 1991 .

[22]  Anwar Elwalid,et al.  Fluid models for the analysis and design of statistical multiplexing with loss priorities on multiple classes of bursty traffic , 1992, [Proceedings] IEEE INFOCOM '92: The Conference on Computer Communications.

[23]  Harold J. Kushner,et al.  Heavy Traffic Analysis of a Data Transmission System with Many Independent Sources , 1993, SIAM J. Appl. Math..

[24]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[25]  Tyrone E. Duncan,et al.  Numerical Methods for Stochastic Control Problems in Continuous Time (Harold J. Kushner and Paul G. Dupuis) , 1994, SIAM Rev..