Numerical Solutions to the Bellman Equation of Optimal Control

In this paper, we present a numerical algorithm to compute high-order approximate solutions to Bellman’s dynamic programming equation that arises in the optimal stabilization of discrete-time nonlinear control systems. The method uses a patchy technique to build local Taylor polynomial approximations defined on small domains, which are then patched together to create a piecewise smooth approximation. The numerical domain is dynamically computed as the level sets of the value function are propagated in reverse time under the closed-loop dynamics. The patch domains are constructed such that their radial boundaries are contained in the level sets of the value function and their lateral boundaries are constructed as invariant sets of the closed-loop dynamics. To minimize the computational effort, an adaptive subdivision algorithm is used to determine the number of patches on each level set depending on the relative error in the dynamic programming equation. Numerical tests in 2D and 3D are given to illustrate the accuracy of the method.

[1]  P. Brunovský.  On optimal stabilization of nonlinear systems , 1967 .

[2]  A. Krener,et al.  Patchy Solutions of Hamilton-Jacobi-Bellman Partial Differential Equations , 2007 .

[3]  Harvey Thomas Banks,et al.  Feedback Control Methodologies for Nonlinear Systems , 2000 .

[4]  Marizio Falcone,et al.  Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations , 1994 .

[5]  Randal W. Beard,et al.  Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation , 1997, Autom..

[6]  P. Frederickson,et al.  Icosahedral Discretization of the Two-Sphere , 1985 .

[7]  D. Lukes.  Optimal Regulation of Nonlinear Dynamical Systems , 1969 .

[8]  I. Norman Katz,et al.  An Iterative Algorithm for Solving Hamilton-Jacobi Type Equations , 2000, SIAM J. Sci. Comput..

[9]  Ruey-Wen Liu,et al.  Construction of Suboptimal Control Sequences , 1967 .

[10]  Carmeliza Luna Navasca.  Local Solutions of the Dynamic Programming Equations and the Hamilton Jacobi Bellman PDE , 2002 .

[11]  F. Ancona,et al.  Patchy Vector Fields and Asymptotic Stabilization , 1999 .

[12]  Arjan van der Schaft,et al.  Analytical Approximation Methods for the Stabilizing Solution of the Hamilton–Jacobi Equation , 2008, IEEE Transactions on Automatic Control.

[13]  Richard Bellman,et al.  Introduction to the mathematical theory of control processes , 1967 .

[14]  Maurizio Falcone,et al.  A Patchy Dynamic Programming Scheme for a Class of Hamilton-Jacobi-Bellman Equations , 2011, SIAM J. Sci. Comput..

[15]  E. G. Al'brekht.  On the optimal stabilization of nonlinear systems , 1961 .

[16]  P. Holmes,et al.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields , 1983, Applied Mathematical Sciences.

[17]  P. Kokotovic,et al.  Sufficient conditions for stabilization of sampled-data nonlinear systems via discrete-time approximations , 1999 .

[18]  H. Ishii,et al.  Approximate solutions of the bellman equation of deterministic control theory , 1984 .

[19]  Kenneth A. Loparo,et al.  Quadratic regulatory theory for analytic non-linear systems with additive controls , 1989, Autom..

[20]  A. Laub.  A schur method for solving algebraic Riccati equations , 1978, 1978 IEEE Conference on Decision and Control including the 17th Symposium on Adaptive Processes.

[21]  E. Gilbert,et al.  Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: Stability and moving-horizon approximations , 1988 .

[22]  Victor M. Becerra,et al.  Optimal control , 2008, Scholarpedia.

[23]  William L. Garrard,et al.  Design of nonlinear automatic flight control systems , 1977, Autom..

[24]  R. Newcomb.  VISCOSITY SOLUTIONS OF HAMILTON-JACOBI EQUATIONS , 2010 .

[25]  Dragan Nesic,et al.  Optimization-Based Stabilization of Sampled-Data Nonlinear Systems via Their Approximate Discrete-Time Models , 2003, SIAM J. Control. Optim..

[26]  Alessandro Chiuso,et al.  Modeling, Estimation and Control , 2007 .

[27]  Yasuaki Oishi,et al.  40th IEEE Conference on Decision and Control , 2002 .

[28]  Paul S. Heckbert,et al.  Using particles to sample and control implicit surfaces , 1994, SIGGRAPH Courses.

[29]  M. Sain,et al.  Series solution of a class of nonlinear optimal regulators , 1996 .

[30]  J. Cloutier,et al.  Control designs for the nonlinear benchmark problem via the state-dependent Riccati equation method , 1998 .

[31]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods , 1999 .

[32]  D. Nesic,et al.  Backstepping on the Euler approximate model for stabilization of sampled-data nonlinear systems , 2001, Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228).