A survey of Markov decision models for control of networks of queues

We review models for the optimal control of networks of queues. Our main emphasis is on models based on Markov decision theory and the characterization of the structure of optimal control policies.

[1]  G. J. Foschini,et al.  A Basic Dynamic Routing Problem and Diffusion , 1978, IEEE Trans. Commun..

[2]  T. M. Farrar Optimal use of an extra server in a two station tandem queueing network , 1993 .

[3]  Tak-Shing Peter Yum,et al.  The Join-Biased-Queue Rule and Its Application to Routing in Computer Communication Networks , 1981, IEEE Trans. Commun..

[4]  Mou-Hsiung Chang Martingale dynamics and optimal routing in a network , 1988 .

[5]  C. N. Laws,et al.  Dynamic Scheduling of a Four-Station Queueing Network , 1990, Probability in the Engineering and Informational Sciences.

[6]  Jr. Shaler Stidham Optimal control of admission to a queueing system , 1985 .

[7]  Linn I. Sennott,et al.  Average Cost Optimal Stationary Policies in Infinite State Markov Decision Processes with Unbounded Costs , 1989, Oper. Res..

[8]  D. Daley,et al.  Light traffic approximations in many-server queues , 1992, Advances in Applied Probability.

[9]  R. Hartley,et al.  Optimisation Over Time: Dynamic Programming and Stochastic Control: , 1983 .

[10]  Richard F. Serfozo,et al.  Optimality of routing and servicing in dependent parallel processing systems , 1991, Queueing Syst. Theory Appl..

[11]  Lawrence M. Wein,et al.  MONOTONE CONTROL OF QUEUEING AND PRODUCTION/INVENTORY SYSTEMS , 1991 .

[12]  K. Krishnan,et al.  Joining the right queue: A Markov decision-rule , 1987, 26th IEEE Conference on Decision and Control.

[13]  P. Whittle Multi‐Armed Bandits and the Gittins Index , 1980 .

[14]  P. R. Kumar,et al.  Optimal control of a queueing system with two heterogeneous servers , 1984 .

[15]  E. J. Collins,et al.  THE JOB SEARCH PROBLEM AS AN EMPLOYER-CANDIDATE GAME , 1990 .

[16]  V. Borkar Control of Markov chains with long-run average cost criterion: the dynamic programming equations , 1989 .

[17]  Shaler Stidham,et al.  Scheduling, Routing, and Flow Control in Stochastic Networks , 1988 .

[18]  R. Cavazos-Cadena Recent results on conditions for the existence of average optimal stationary policies , 1991 .

[19]  J. Michael Harrison,et al.  Brownian Models of Queueing Networks with Heterogeneous Customer Populations , 1988 .

[20]  Armand M. Makowski,et al.  K competing queues with geometric service requirements and linear costs: The μc-rule is always optimal☆ , 1985 .

[21]  Jean Walrand,et al.  Extensions of the multiarmed bandit problem: The discounted case , 1985 .

[22]  J. Ben Atkinson,et al.  An Introduction to Queueing Networks , 1988 .

[23]  Gordon M. Bull,et al.  Time-sharing systems , 1971 .

[24]  Philippe Nain,et al.  Optimal scheduling in some multiqueue single-server systems , 1992 .

[25]  Timothy Martin Farrar Resource allocation in systems of queues , 1992 .

[26]  Dimitri P. Bertsekas,et al.  Dynamic Programming: Deterministic and Stochastic Models , 1987 .

[27]  Vivek S. Borkar,et al.  Controlled Markov Chains and Stochastic Networks , 1983 .

[28]  Philippe Nain,et al.  Optimal scheduling in some multi-queue single-server systems , 1990, Proceedings. IEEE INFOCOM '90: Ninth Annual Joint Conference of the IEEE Computer and Communications Societies@m_The Multiple Facets of Integration.

[29]  Tapani Lehtonen,et al.  On the optimality of the shortest line discipline , 1984 .

[30]  Vidyadhar G. Kulkarni,et al.  Optimal control of two parallel infinite-server queues , 1990, 29th IEEE Conference on Decision and Control.

[31]  Ronald Menich,et al.  Optimally of shortest queue routing for dependent service stations , 1987, 26th IEEE Conference on Decision and Control.

[32]  Lawrence M. Wein,et al.  Scheduling Networks of Queues: Heavy Traffic Analysis of a Two-Station Closed Network , 1990, Oper. Res..

[33]  D. J. Houck,et al.  Comparison of Policies for Routing Customers to Parallel Queueing Systems , 1987, Oper. Res..

[34]  Manfred SchÄl,et al.  Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal , 1975 .

[35]  J. Gittins Bandit processes and dynamic allocation indices , 1979 .

[36]  R. Cavazos-Cadena Necessary conditions for the optimality equation in average-reward Markov decision processes , 1989 .

[37]  Ger Koole,et al.  On the Optimality of the Generalized Shortest Queue Policy , 1990, Probability in the Engineering and Informational Sciences.

[38]  D. Teneketzis,et al.  Routing in queueing networks under imperfect information: stochastic dominance and thresholds , 1989 .

[39]  R. Weber,et al.  Optimal control of service rates in networks of queues , 1987, Advances in Applied Probability.

[40]  David D. Yao,et al.  IN A MULTI-CLASS QUEUEING NETWORK: RESULTS AND CONJECTURES , 1990 .

[41]  B. Hajek Optimal control of two interacting service stations , 1982, 1982 21st IEEE Conference on Decision and Control.

[42]  Shaler Stidham,et al.  Control of arrivals to two queues in series , 1985 .

[43]  M. K. Ghosh,et al.  Discrete-time controlled Markov processes with average cost criterion: a survey , 1993 .

[44]  Dong-Wan Tcha,et al.  Optimal Control of Single-Server Queuing Networks and Multi-Class M/G/1 Queues with Feedback , 1977, Oper. Res..

[45]  Jean Walrand,et al.  A note on optimal control of a queuing system with two heterogeneous servers , 1984 .

[46]  Linn I. Sennott,et al.  Average Cost Semi-Markov Decision Processes and the Control of Queueing Systems , 1989, Probability in the Engineering and Informational Sciences.

[47]  J. Harrison,et al.  Brownian motion and stochastic flow systems , 1986 .

[48]  Vivek S. Borkar,et al.  Control of Markov Chains with Long-Run Average Cost Criterion , 1988 .

[49]  Hong Chen,et al.  Control and scheduling in a two-station queueing network: Optimal policies and heuristics , 1994, Queueing Syst. Theory Appl..

[50]  U. Ozguner,et al.  On decentralized dynamic routing for congested traffic networks , 1982 .

[51]  Donald M. Topkis,et al.  Minimizing a Submodular Function on a Lattice , 1978, Oper. Res..

[52]  P. Nain,et al.  Interchange arguments for classical scheduling problems in queues , 1989 .

[53]  J. Michael Harrison,et al.  Dynamic Scheduling of a Multiclass Queue: Discount Optimality , 1975, Oper. Res..

[54]  T. B. Crabill Optimal Control of a Service Facility with Variable Exponential Service Times and Constant Arrival Rate , 1972 .

[55]  Scott Shenker,et al.  The Optimal Control of Heterogeneous Queueing Systems: A Paradigm for Load-Sharing and Routing , 1989, IEEE Trans. Computers.

[56]  J. Walrand,et al.  Interchange arguments in stochastic scheduling , 1989 .

[57]  J. Michael Harrison,et al.  A Priority Queue with Discounted Linear Costs , 1975, Oper. Res..

[58]  Anthony Ephremides,et al.  A simple dynamic routing problem , 1980 .

[59]  J. Baras,et al.  Two competing queues with linear costs: the μc-rule is often optimal , 1984, Advances in Applied Probability.

[60]  Ward Whitt,et al.  Deciding Which Queue to Join: Some Counterexamples , 1986, Oper. Res..

[61]  Bruce E. Hajek,et al.  Extremal Splittings of Point Processes , 1985, Math. Oper. Res..

[62]  R. Weber On the optimal assignment of customers to parallel servers , 1978, Journal of Applied Probability.

[63]  Pravin K. Johri,et al.  Optimality of the shortest line discipline with state-dependent service rates , 1989 .

[64]  Peter Whittle,et al.  Optimization Over Time , 1982 .

[65]  Jean C. Walrand,et al.  An introduction to queueing networks , 1989, Prentice Hall International editions.

[66]  David D. Yao,et al.  Monotone Optimal Control of Permutable GSMPs , 1994, Math. Oper. Res..

[67]  Wayne L. Winston OPTIMALITY OF THE SHORTEST LINE DISCIPLINE , 1977 .