Exit time risk-sensitive control for systems of cooperative agents

AbstractWe study a sequence of many-agent exit time stochastic control problems, parameterized by the number of agents, with risk-sensitive cost structure. We identify a fully characterizing assumption, under which each such control problem corresponds to a risk-neutral stochastic control problem with additive cost, and sequentially to a risk-neutral stochastic control problem on the simplex that retains only the distribution of states of agents, while discarding further specific information about the state of each agent. Under some additional assumptions, we also prove that the sequence of value functions of these stochastic control problems converges to the value function of a deterministic control problem, which can be used for the design of nearly optimal controls for the original problem, when the number of agents is sufficiently large.

[1]  Ian R. Petersen,et al.  Robust Properties of Risk-Sensitive Control , 2000, Math. Control. Signals Syst..

[2]  M. K. Ghosh,et al.  Discrete-time controlled Markov processes with average cost criterion: a survey , 1993 .

[3]  G. Kallianpur.  Stochastic differential equations and diffusion processes , 1981 .

[4]  Anna Ja 'skiewicz.  Average optimality for risk-sensitive control with general state space , 2007 .

[5]  Anja Vogler,et al.  Continuous Time Markov Decision Processes Theory And Applications , 2016 .

[6]  M. Manhart,et al.  Markov Processes , 2018, Introduction to Stochastic Processes and Simulation.

[7]  Peter Whittle,et al.  Optimal Control: Basics and Beyond , 1996 .

[8]  P. Dupuis,et al.  Minimax optimal control of stochastic uncertain systems with relative entropy constraints , 1997, Proceedings of the 36th IEEE Conference on Decision and Control.

[9]  O. Hernández-Lerma,et al.  Further topics on discrete-time Markov control processes , 1999 .

[10]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[11]  J. Norris.  Appendix: probability and measure , 1997 .

[12]  Andrzej Ruszczynski,et al.  Risk-averse dynamic programming for Markov decision processes , 2010, Math. Program..

[13]  W. Fleming,et al.  Risk-Sensitive Control of Finite State Machines on an Infinite Horizon I , 1997 .

[14]  M. K. Ghosh,et al.  Risk-sensitive control of continuous time Markov chains , 2014, 1409.4032.

[15]  E. Fernandez-Gaucherand,et al.  Controlled Markov chains with exponential risk-sensitive criteria: modularity, structured policies and applications , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[16]  Sean P. Meyn,et al.  Risk-Sensitive Optimal Control for Markov Decision Processes with Monotone Cost , 2002, Math. Oper. Res..

[17]  P. Dupuis,et al.  Large Deviation Principle For Finite-State Mean Field Interacting Particle Systems , 2016, 1601.06219.

[18]  M. Sion.  On general minimax theorems , 1958 .

[19]  M. J. Sobel,et al.  Discounted MDP's: distribution functions and exponential utility maximization , 1987 .

[20]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[21]  R. Howard,et al.  Risk-Sensitive Markov Decision Processes , 1972 .

[22]  Rolando Cavazos-Cadena,et al.  Optimality equations and inequalities in a class of risk-sensitive average cost Markov decision chains , 2010, Math. Methods Oper. Res..

[23]  L. Rogers.  Stochastic differential equations and diffusion processes: Nobuyuki Ikeda and Shinzo Watanabe North-Holland, Amsterdam, 1981, xiv + 464 pages, Dfl.175.00 , 1982 .

[24]  William M. McEneaney,et al.  Risk-Sensitive and Robust Escape Criteria , 1997 .

[25]  S. Marcus,et al.  Risk sensitive control of Markov processes in countable state space , 1996 .

[26]  W. Fleming,et al.  Asymptotic expansions for Markov processes with lévy generators , 1989 .

[27]  Lukasz Stettner,et al.  Infinite Horizon Risk Sensitive Control of Discrete Time Markov Processes under Minorization Property , 2007, SIAM J. Control. Optim..