Relative Time and Stochastic Control With Non-Smooth Features

The stochastic calculus of non-smooth functions indicates that for a continuous semi-martingale <inline-formula> <tex-math notation="LaTeX">$X(t)$</tex-math></inline-formula>, the changes of a function <inline-formula> <tex-math notation="LaTeX">$h[X(t)]$</tex-math></inline-formula> at its semi-smooth point (both right- and left-hand side derivatives exist) <inline-formula> <tex-math notation="LaTeX">$X(t) = x$</tex-math></inline-formula> in <inline-formula> <tex-math notation="LaTeX">$[t,t + dt]$</tex-math></inline-formula> is at the scale of the local time of <inline-formula> <tex-math notation="LaTeX">$X(t)$</tex-math></inline-formula>, with a mean of the order <inline-formula> <tex-math notation="LaTeX">$\sqrt{dt}$</tex-math></inline-formula> in the case of Ito processes. We introduce the <bold><italic>relative time</italic></bold> which evolves at the scale of local time when the semi-martingale is at a semi-smooth point of <inline-formula> <tex-math notation="LaTeX">$h(x)$</tex-math></inline-formula>. The change of <inline-formula> <tex-math notation="LaTeX">$h[X(t)]$</tex-math></inline-formula> in <inline-formula> <tex-math notation="LaTeX">$[t,t + dt]$</tex-math></inline-formula> can be precisely measured in the scale of relative time, while this change is wrongly ignored with regular time scale <inline-formula> <tex-math notation="LaTeX">$dt$</tex-math></inline-formula>. The optimal control problem is well defined with the regular time replaced by the relative time; however, dynamic programming does not seem work well for this problem. We apply the direct-comparison-based optimization approach to the control problem formulated in relative time and derive the generalized Hamilton-Jacobi-Bellman (HJB) equations, which consist of two parts, the classical HJB equation for smooth points, and some additional relations for semi-smooth points. Under some bounded conditions, the optimal value function is the (classical) solution to the generalized HJB equations, and viscosity solution is not needed. In addition, we show that the singular control problem can be formulated and solved in the same framework with the relative time.

[1]  Hiroshi Tanaka.  Stochastic differential equations with reflecting boundary condition in convex regions , 1979 .

[2]  G. Kallianpur.  Stochastic differential equations and diffusion processes , 1981 .

[3]  P. Lions,et al.  Viscosity solutions of Hamilton-Jacobi equations , 1983 .

[4]  G. Barles,et al.  Discontinuous solutions of deterministic optimal stopping time problems , 1987 .

[5]  E. Barron,et al.  Semicontinuous Viscosity Solutions For Hamilton–Jacobi Equations With Convex Hamiltonians , 1990 .

[6]  W. Fleming,et al.  Controlled Markov processes and viscosity solutions , 1992 .

[7]  A. Shiryayev,et al.  Quadratic covariation and an extension of Itô's formula , 1995 .

[8]  X. Zhou,et al.  Stochastic Controls: Hamiltonian Systems and HJB Equations , 1999 .

[9]  Nathalie Eisenbaum.  Integration with respect to local time , 2000 .

[10]  G. Peskir.  A Change-of-Variable Formula with Local Time on Curves , 2005 .

[11]  Xi-Ren Cao,et al.  Stochastic learning and optimization - A sensitivity-based approach , 2007, Annual Reviews in Control.

[12]  Martin Herdegen.  Optimal Stopping and Applications Example 2 : American options , 2009 .

[13]  Sheng Miao,et al.  Smooth Value Functions for a Class of Nonsmooth Utility Maximization Problems , 2010, SIAM J. Financial Math..

[14]  Tao Lu,et al.  Stochastic control via direct comparison , 2011, Discret. Event Dyn. Syst..

[15]  Bruno H. Strulovici,et al.  On the Smoothness of Value Functions and the Existence of Optimal Strategies , 2012 .

[16]  Yuan-Hua Ni,et al.  Policy Iteration Algorithm for Singular Controlled Diffusion Processes , 2013, SIAM J. Control. Optim..

[17]  Li Qiu,et al.  Partial-Information State-Based Optimization of Partially Observable Markov Decision Processes and the Separation Principle , 2014, IEEE Transactions on Automatic Control.

[18]  Xi-Ren Cao,et al.  Optimization of Average Rewards of Time Nonhomogeneous Markov Chains , 2015, IEEE Transactions on Automatic Control.

[19]  B. Kawohl,et al.  Jump discontinuous viscosity solutions to second order degenerate elliptic equations , 2015 .

[20]  Anja Walter,et al.  Introduction To Stochastic Calculus With Applications , 2016 .

[21]  S. Aachen.  Stochastic Differential Equations An Introduction With Applications , 2016 .

[22]  Xi-Ren Cao,et al.  SENSITIVITY ANALYSIS OF NONLINEAR BEHAVIOR WITH DISTORTED PROBABILITY , 2013 .

[23]  B. Øksendal,et al.  Applied Stochastic Control of Jump Diffusions , 2004, Universitext.