Stochastic Differential Games: A Sampling Approach via FBSDEs

The aim of this work is to present a sampling-based algorithm designed to solve various classes of stochastic differential games. The foundation of the proposed approach lies in the formulation of the game solution in terms of a decoupled pair of forward and backward stochastic differential equations (FBSDEs). In light of the nonlinear version of the Feynman–Kac lemma, probabilistic representations of solutions to the nonlinear Hamilton–Jacobi–Isaacs equations that arise for each class are obtained. These representations are in form of decoupled systems of FBSDEs, which may be solved numerically.

[1]  B. Øksendal.  Stochastic Differential Equations , 1985 .

[2]  B. Øksendal.  Stochastic differential equations : an introduction with applications , 1987 .

[3]  J. Yong,et al.  Solving forward-backward stochastic differential equations explicitly — a four step scheme , 1994 .

[4]  William M. McEneaney,et al.  A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs , 2007, SIAM J. Control. Optim..

[5]  Robert Denk,et al.  A forward scheme for backward SDEs , 2007 .

[6]  Adam Krzyzak,et al.  A Distribution-Free Theory of Nonparametric Regression , 2002, Springer series in statistics.

[7]  Evangelos Theodorou,et al.  Stochastic optimal control via forward and backward stochastic differential equations and importance sampling , 2018, Autom..

[8]  J. E. Potter,et al.  Fuel Optimal Reorientation of Axisymmetric Spacecraft , 1970 .

[9]  Sertac Karaman,et al.  Efficient High-Dimensional Stochastic Optimal Motion Control using Tensor-Train Decomposition , 2015, Robotics: Science and Systems.

[10]  J. Douglas,et al.  Numerical methods for forward-backward stochastic differential equations , 1996 .

[11]  L. Györfi,et al.  A Distribution-Free Theory of Nonparametric Regression (Springer Series in Statistics) , 2002 .

[12]  E. Gobet,et al.  Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations , 2006 .

[13]  Hai-ping Shi.  Backward stochastic differential equations in finance , 2010 .

[14]  Hans Seywald,et al.  Minimum fuel spacecraft reorientation , 1994 .

[15]  Jianfeng Zhang.  Backward Stochastic Differential Equations , 2017 .

[16]  Jianfeng Zhang.  A numerical scheme for BSDEs , 2004 .

[17]  G. N. Milstein,et al.  Numerical Algorithms for Forward-Backward Stochastic Differential Equations , 2006, SIAM J. Sci. Comput..

[18]  Nizar Touzi,et al.  A Probabilistic Numerical Method for Fully Nonlinear Parabolic PDEs , 2009, 0905.1863.

[19]  Joel W. Burdick,et al.  Semidefinite relaxations for stochastic optimal control policies , 2014, 2014 American Control Conference.

[20]  Ioannis Karatzas,et al.  Brownian Motion and Stochastic Calculus , 1987 .

[21]  E. Todorov,et al.  Linearly Solvable Optimal Control , 2013 .

[22]  Chris P. Tsokos,et al.  Stochastic Differential Games. Theory and Applications , 2012 .

[23]  J. Lepeltier,et al.  Existence for BSDE with superlinear–quadratic coefficient , 1998 .

[24]  Jun Morimoto,et al.  Minimax Differential Dynamic Programming: An Application to Robust Biped Walking , 2002, NIPS.

[25]  Harold J. Kushner,et al.  On stochastic differential games: Sufficient conditions that a given strategy be a saddle point, and numerical procedures for the solution of the game☆ , 1969 .

[26]  Stefan Schaal,et al.  A Generalized Path Integral Control Approach to Reinforcement Learning , 2010, J. Mach. Learn. Res..

[27]  X. Zhou,et al.  Stochastic Controls: Hamiltonian Systems and HJB Equations , 1999 .

[28]  Avner Friedman,et al.  Stochastic differential games , 1972 .

[29]  Francis A. Longstaff,et al.  Valuing American Options by Simulation: A Simple Least-Squares Approach , 2001 .

[30]  Joel W. Burdick,et al.  Linear Hamilton Jacobi Bellman Equations in high dimensions , 2014, 53rd IEEE Conference on Decision and Control.

[31]  C. Atkeson,et al.  Minimax differential dynamic programming: application to a biped walking robot , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[32]  J. Chassagneux,et al.  Numerical simulation of quadratic BSDEs , 2013, 1307.5741.

[33]  Tyrone E. Duncan,et al.  Some stochastic differential games with state dependent noise , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[34]  S. Kahne,et al.  Optimal control: An introduction to the theory and ITs applications , 1967, IEEE Transactions on Automatic Control.

[35]  G. Barles,et al.  Convergence of approximation schemes for fully nonlinear second order equations , 1990, 29th IEEE Conference on Decision and Control.

[36]  Bruno Bouchard,et al.  Discrete-Time Approximation of BSDEs and Probabilistic Schemes for Fully Nonlinear PDEs , 2009 .

[37]  W. Fleming,et al.  Controlled Markov processes and viscosity solutions , 1992 .

[38]  D. Xiu.  Numerical Methods for Stochastic Computations: A Spectral Method Approach , 2010 .

[39]  L. Berkovitz,et al.  A Variational Approach to Differential Games , 1960 .

[40]  Jie Shen,et al.  On Numerical Approximations of Forward-Backward Stochastic Differential Equations , 2008, SIAM J. Numer. Anal..

[41]  Emmanuel Trélat,et al.  Nonlinear Optimal Control via Occupation Measures and LMI-Relaxations , 2007, SIAM J. Control. Optim..

[42]  HAROLD J. KUSHNER,et al.  Numerical Approximations for Stochastic Differential Games , 2002, SIAM J. Control. Optim..

[43]  Daniel E. Quevedo,et al.  Maximum Hands-Off Control: A Paradigm of Control Effort Minimization , 2014, IEEE Transactions on Automatic Control.

[44]  Daniel E. Quevedo,et al.  Maximum hands-off control and L1 optimality , 2013, 52nd IEEE Conference on Decision and Control.

[45]  Emmanuel Gobet,et al.  Error expansion for the discretization of backward stochastic differential equations , 2006, math/0602503.

[46]  Y. Ho,et al.  Differential games and optimal pursuit-evasion strategies , 1965 .

[47]  Zhimin Zhang,et al.  Numerical Solutions for Stochastic Differential Games With Regime Switching , 2008, IEEE Transactions on Automatic Control.

[48]  J. Ma,et al.  Forward-Backward Stochastic Differential Equations and their Applications , 2007 .

[49]  B. Bouchard,et al.  Discrete-time approximation and Monte-Carlo simulation of backward stochastic differential equations , 2004 .

[50]  Wei Sun,et al.  Game Theoretic continuous time Differential Dynamic Programming , 2015, 2015 American Control Conference (ACC).

[51]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[52]  Freddy Delbaen,et al.  On the uniqueness of solutions to quadratic BSDEs with convex generators and unbounded terminal conditions: the critical case , 2013 .

[53]  P. Kloeden,et al.  Numerical Solution of Stochastic Differential Equations , 1992 .

[54]  Randal W. Beard,et al.  Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation , 1997, Autom..

[55]  Francesca Da Lio,et al.  Uniqueness Results for Second-Order Bellman--Isaacs Equations under Quadratic Growth Assumptions and Applications , 2010, SIAM J. Control. Optim..

[56]  Juan Li,et al.  Stochastic Differential Games and Viscosity Solutions of Hamilton--Jacobi--Bellman--Isaacs Equations , 2008, SIAM J. Control. Optim..

[57]  M. Kobylanski.  Backward stochastic differential equations and partial differential equations with quadratic growth , 2000 .

[58]  J. Lepeltier,et al.  Zero-sum stochastic differential games and backward equations , 1995 .