A max-plus based randomized algorithm for solving a class of HJB PDEs

McEneaney introduced the curse of dimensionality free method for the special class of infinite horizon optimal control problems where the Hamiltonian is represented as a maximum of quadratic affine functions. This method is featured by its cubic complexity with respect to the state space dimension, but the number of basis functions is multiplied by the number of switches at each iteration, referred to as the `curse of complexity'. In previous works, an SDP-based pruning technique was incorporated into the method in order to reduce the curse of complexity. Its efficiency was proved on many examples. In this paper we develop a new max-plus based randomized algorithm to solve the same class of infinite horizon optimal control problems. The major difference between the new algorithm and the previous SDP-based curse of dimensionality free method is that, instead of adding a large number of functions and then pruning the less useful ones, the new algorithm finds in cheap computation time (linear in the current number of basis functions), by a randomized procedure, useful quadratic functions and adds only those functions to the set of basis functions. Experimental results show that the max-plus randomized algorithm can reach the same precision order obtained by the SDP-based method with a speedup varying from 10 up to 100 and that the maximal precision order attainable by the new algorithm is much better than what can be done by the SDP-based algorithm in reasonable computation time. Besides, with the randomized algorithm we are now able to tackle switched problems with more number of switches, which will allow us to extend the algorithm to more general classes of optimal control problems.

[1]  William M. McEneaney,et al.  A max-plus based fundamental solution for a class of infinite dimensional Riccati equations , 2011, IEEE Conference on Decision and Control and European Control Conference.

[2]  Stéphane Gaubert,et al.  The Max-Plus Finite Element Method for Solving Deterministic Optimal Control Problems: Basic Properties and Convergence Analysis , 2008, SIAM J. Control. Optim..

[3]  P. Lions,et al.  Two approximations of solutions of Hamilton-Jacobi equations , 1984 .

[4]  Michael L. Littman,et al.  Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes , 1997, UAI.

[5]  William M. McEneaney,et al.  Reduced-complexity numerical method for optimal gate synthesis , 2010, 1011.6013.

[6]  Zheng Qu.  Nonlinear Perron-Frobenius theory and max-plus numerical methods for Hamilton-Jacobi equations , 2013 .

[7]  Srinivas Sridharan.  Deterministic filtering and max-plus methods for robust state estimation in multi-sensor settings , 2012, 1211.1449.

[8]  William M. McEneaney,et al.  A Max-Plus-Based Algorithm for a Hamilton--Jacobi--Bellman Equation of Nonlinear Filtering , 2000, SIAM J. Control. Optim..

[9]  William M. McEneaney,et al.  A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs , 2007, SIAM J. Control. Optim..

[10]  William M. McEneaney,et al.  Curse of dimensionality reduction in max-plus based approximation methods: Theoretical estimates and improved pruning algorithms , 2011, IEEE Conference on Decision and Control and European Control Conference.

[11]  Marizio Falcone,et al.  Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations , 1994 .

[12]  M. Falcone.  A numerical approach to the infinite horizon problem of deterministic control theory , 1987 .

[13]  Weihong Zhang,et al.  Speeding Up the Convergence of Value Iteration in Partially Observable Markov Decision Processes , 2011, J. Artif. Intell. Res..

[14]  William M. McEneaney,et al.  Max-plus methods for nonlinear control and estimation , 2005 .

[15]  Zheng Qu,et al.  Contraction of Riccati flows applied to the convergence analysis of the max-plus curse of dimensionality free method , 2013, 2013 European Control Conference (ECC).

[16]  W.M. McEneaney,et al.  Curse-of-complexity attenuation in the curse-of-dimensionality-free method for HJB PDEs , 2008, 2008 American Control Conference.