Risk-Averse Control of Undiscounted Transient Markov Models

We use Markov risk measures to formulate a risk-averse version of the undiscounted total cost problem for a transient controlled Markov process. Using the new concept of a multikernel, we derive conditions for a system to be risk transient, that is, to have finite risk over an infinite time horizon. We derive risk-averse dynamic programming equations satisfied by the optimal policy and we describe methods for solving these equations. We illustrate the results on an optimal stopping problem and an organ transplantation problem.

[1]  Johannes Leitner.  A SHORT NOTE ON SECOND‐ORDER STOCHASTIC DOMINANCE PRESERVING COHERENT RISK MEASURES , 2005 .

[2]  S. C. Jaquette.  A Utility Criterion for Markov Decision Processes , 1976 .

[3]  A. Ruszczynski,et al.  Optimization of Risk Measures , 2006 .

[4]  Manfred SchÄl,et al.  Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal , 1975 .

[5]  Jeffrey P. Kharoufeh,et al.  Monotone optimal replacement policies for a Markovian deteriorating system in a controllable environment , 2010, Oper. Res. Lett..

[6]  Laurent El Ghaoui,et al.  Robust Control of Markov Decision Processes with Uncertain Transition Matrices , 2005, Oper. Res..

[7]  Wlodzimierz Ogryczak,et al.  On consistency of stochastic dominance and mean–semideviation models , 2001, Math. Program..

[8]  R. Howard,et al.  Risk-Sensitive Markov Decision Processes , 1972 .

[9]  John N. Tsitsiklis,et al.  An Analysis of Stochastic Shortest Path Problems , 1991, Math. Oper. Res..

[10]  Joanna L. Y. Ho,et al.  Hope: An empirical study of attitude toward the timing of uncertainty resolution , 1994 .

[11]  F. Delbaen,et al.  Dynamic Monetary Risk Measures for Bounded Discrete-Time Processes , 2004, math/0410453.

[12]  U. Rieder,et al.  Markov Decision Processes with Applications to Finance , 2011 .

[13]  A. S. Manne.  Linear Programming and Sequential Decisions , 1960 .

[14]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .

[15]  Alexander Shapiro,et al.  Conditional Risk Mappings , 2005, Math. Oper. Res..

[16]  Alexander Shapiro,et al.  Optimization of Convex Risk Functions , 2006, Math. Oper. Res..

[17]  L. Breuer.  Introduction to Stochastic Processes , 2022, Statistical Methods for Climate Scientists.

[18]  Stanley R. Pliska.  ON THE TRANSIENT CASE FOR MARKOV DECISION CHAINS WITH GENERAL STATE SPACES , 1978 .

[19]  D. Klatte.  Nonsmooth equations in optimization , 2002 .

[20]  O. Hernández-Lerma,et al.  Further topics on discrete-time Markov control processes , 1999 .

[21]  M. Frittelli,et al.  RISK MEASURES AND CAPITAL REQUIREMENTS FOR PROCESSES , 2006 .

[22]  Andrzej Ruszczynski,et al.  Risk-averse dynamic programming for Markov decision processes , 2010, Math. Program..

[23]  Wlodzimierz Ogryczak,et al.  From stochastic dominance to mean-risk models: Semideviations as risk measures , 1999, Eur. J. Oper. Res..

[24]  Philippe Artzner,et al.  Coherent Measures of Risk , 1999 .

[25]  Uriel G. Rothblum,et al.  Optimal stopping, exponential utility, and linear programming , 1979, Math. Program..

[26]  Wlodzimierz Ogryczak,et al.  Dual Stochastic Dominance and Related Mean-Risk Models , 2002, SIAM J. Optim..

[27]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[28]  Anna Jaskiewicz,et al.  Stochastic Games with Unbounded Payoffs: Applications to Robust Control in Economics , 2011, Dyn. Games Appl..

[29]  Helena Jasiulewicz.  Application of mixture models to approximation of age-at-death distribution , 1997 .

[30]  L. C. M. Kallenberg,et al.  Linear programming and finite Markovian control problems , 1984 .

[31]  Charles S. Tapiero,et al.  A Mean Variance Approach to the Optimal Machine Maintenance and Replacement Problem , 1979 .

[32]  Yoshio Ohtsubo,et al.  Optimal threshold probability in undiscounted Markov decision processes with a target set , 2004, Appl. Math. Comput..

[33]  Andrew J. Schaefer,et al.  The Optimal Timing of Living-Donor Liver Transplantation , 2004, Manag. Sci..

[34]  M. Teboulle,et al.  AN OLD‐NEW CONCEPT OF CONVEX RISK MEASURES: THE OPTIMIZED CERTAINTY EQUIVALENT , 2007 .

[35]  Jonathan Eckstein,et al.  YASAI: Yet Another Add-in for Teaching Elementary Monte Carlo Simulation in Excel , 2002 .

[36]  G. Pflug,et al.  Modeling, Measuring and Managing Risk , 2008 .

[37]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[38]  A. Shwartz,et al.  Handbook of Markov decision processes : methods and applications , 2002 .

[39]  David Heath,et al.  Coherent multiperiod risk adjusted values and Bellman’s principle , 2007, Ann. Oper. Res..

[40]  Erhan Çinlar,et al.  Introduction to stochastic processes , 1974 .

[41]  Susanne Klöppel,et al.  DYNAMIC INDIFFERENCE VALUATION VIA CONVEX RISK MEASURES , 2007 .

[42]  John N. Tsitsiklis,et al.  Mean-Variance Optimization in Markov Decision Processes , 2011, ICML.

[43]  D. J. White,et al.  A Survey of Applications of Markov Decision Processes , 1993 .

[44]  D. D. Sworder,et al.  Minimax Control of Discrete Time Stochastic Systems , 1964 .

[45]  Stephen D. Patek,et al.  On terminating Markov decision processes with a risk-averse objective function , 2001, Autom..

[46]  Dimitri P. Bertsekas,et al.  Stochastic optimal control : the discrete time case , 2007 .

[47]  Yoshio Ohtsubo.  Minimizing risk models in stochastic shortest path problems , 2003, Math. Methods Oper. Res..

[48]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[49]  Onésimo Hernández-Lerma,et al.  Controlled Markov Processes , 1965 .

[50]  Steven D. Levitt,et al.  On Modeling Risk in Markov Decision Processes , 2001 .

[51]  Y. Nie,et al.  Shortest path problem considering on-time arrival probability , 2009 .

[52]  Lodewijk C. M. Kallenberg.  Survey of linear programming for standard and nonstandard Markovian control problems. Part II: Applications , 1994, Math. Methods Oper. Res..

[53]  R. Rockafellar,et al.  Conditional Value-at-Risk for General Loss Distributions , 2001 .

[54]  Lyn C. Thomas,et al.  Modelling the profitability of credit cards by Markov decision processes , 2011, Eur. J. Oper. Res..

[55]  R. Bellman.  Dynamic programming. , 1957, Science.

[56]  Stella X. Yu,et al.  Optimization Models for the First Arrival Target Distribution Function in Discrete Time , 1998 .

[57]  E. Denardo.  On Linear Programming in a Markov Decision Problem , 1970 .

[58]  A. F. Veinott.  Discrete Dynamic Programming with Sensitive Discount Optimality Criteria , 1969 .

[59]  Frank Riedel,et al.  Dynamic Coherent Risk Measures , 2003 .

[60]  R. Tyrrell Rockafellar.  Conjugate Duality and Optimization , 1974 .

[61]  J. Wessels.  Markov programming by successive approximations by respect to weighted supremum norms , 1976, Advances in Applied Probability.

[62]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.