Optimal impulsive control of piecewise deterministic Markov processes

In this paper, we study the infinite-horizon expected discounted continuous-time optimal control problem for Piecewise Deterministic Markov Processes with both impulsive and gradual (also called continuous) controls. The set of admissible control strategies is supposed to be formed by policies possibly randomized and depending on the past-history of the process. We assume that the gradual control acts on the jump intensity and on the transition measure, but not on the flow. The so-called Hamilton–Jacobi–Bellman (HJB) equation associated to this optimization problem is analyzed. We provide sufficient conditions for the existence of a solution to the HJB equation and show that the solution is in fact unique and coincides with the value function of the control problem. Moreover, the existence of an optimal control strategy is proven having the property to be stationary and non-randomized.

[1]  M. Schäl.  On dynamic programming: Compactness of the space of policies , 1975 .

[2]  J. Jacod.  Calcul stochastique et problèmes de martingales , 1979 .

[3]  Robert J. Elliott,et al.  Stochastic calculus and applications , 1984, IEEE Transactions on Automatic Control.

[4]  D. Sworder.  Stochastic calculus and applications , 1984, IEEE Transactions on Automatic Control.

[5]  M. K rn,et al.  Stochastic Optimal Control , 1988 .

[6]  A. A. Yushkevich,et al.  Verification Theorems for Markov Decision Processes with Controlled Deterministic Drift and Gradual and Impulsive Controls , 1990 .

[7]  P. Malliavin.  Infinite dimensional analysis , 1993 .

[8]  Jane J. Ye,et al.  Impulse Control of Piecewise Deterministic Markov Processes , 1995 .

[9]  Mark H. Davis.  Markov Models and Optimization , 1995 .

[10]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[11]  O. Costa,et al.  Impulse and continuous control of piecewise deterministic Markov processes , 2000 .

[12]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[13]  Oswaldo Luiz do Valle Costa,et al.  Continuous Average Control of Piecewise Deterministic Markov Processes , 2013 .

[14]  Marco Pavone,et al.  Stochastic Optimal Control , 2015 .

[15]  Dan Goreac,et al.  Asymptotic Control for a Class of Piecewise Deterministic Markov Processes Associated to Temperate Viruses , 2015, SIAM J. Control. Optim..

[16]  F. Dufour,et al.  Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach , 2014, Applied Mathematics & Optimization.

[17]  Oswaldo Luiz do Valle Costa,et al.  Constrained and Unconstrained Optimal Control of Piecewise Deterministic Markov Processes , 2015 .

[18]  F. Dufour,et al.  Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach , 2014, 1402.6106.

[19]  Oswaldo Luiz V. Costa,et al.  Constrained and Unconstrained Optimal Discounted Control of Piecewise Deterministic Markov Processes , 2016, SIAM J. Control. Optim..