Approximate dynamic programming for stochastic reachability

In this work we illustrate how approximate dynamic programing can be utilized to address problems of stochastic reachability in infinite state and control spaces. In particular we focus on the reach-avoid problem and approximate the value function on a linear combination of radial basis functions. In this way we get significant computational advantages with which we obtain tractable solutions to problems that cannot be solved via generic space gridding due to the curse of dimensionality. Numerical simulations indicate that control policies coming as a result of approximating the value function of stochastic reachability problems achieve close to optimal performance.

[1]  Benjamin Van Roy,et al.  The Linear Programming Approach to Approximate Dynamic Programming , 2003, Oper. Res..

[2]  John Lygeros,et al.  Approximate dynamic programming via sum of squares programming , 2012, 2013 European Control Conference (ECC).

[3]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[4]  Marco C. Campi,et al.  The Exact Feasibility of Randomized Solutions of Uncertain Convex Programs , 2008, SIAM J. Optim..

[5]  John Lygeros,et al.  Verification of discrete time stochastic hybrid systems: A stochastic reach-avoid decision problem , 2010, Autom..

[6]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[7]  John Lygeros,et al.  On the connections between PCTL and dynamic programming , 2009, HSCC '10.

[8]  M. Campi,et al.  The scenario approach for systems and control design , 2008 .

[9]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .

[10]  W. Fleming Book Review: Discrete-time Markov control processes: Basic optimality criteria , 1997 .

[11]  Milos Hauskrecht,et al.  Solving Factored MDPs with Continuous and Discrete Variables , 2004, UAI.

[12]  John Lygeros,et al.  Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems , 2008, Autom..

[13]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[14]  Stephen P. Boyd,et al.  Approximate dynamic programming via iterated Bellman inequalities , 2015 .

[15]  Milos Hauskrecht,et al.  Linear Program Approximations for Factored Continuous-State Markov Decision Processes , 2003, NIPS.

[16]  Giuseppe Carlo Calafiore,et al.  The scenario approach to robust control design , 2006, IEEE Transactions on Automatic Control.