Lax-Oleinik-type formulas and efficient algorithms for certain high-dimensional optimal control problems

We provide analytical solutions to certain optimal control problems whose running cost depends on the state variable and with constraints on the control. We also provide Lax-Oleinik-type representation formulas for the corresponding Hamilton-Jacobi partial differential equations with statedependent Hamiltonians. Additionally, we present efficient, grid-free numerical solvers based on our representation formulas, which are shown, in some cases, to scale linearly with the state dimension, and thus, to overcome the curse of dimensionality. Using existing optimization methods and the min-plus technique, we extend our numerical solvers to address more general classes of convex and nonconvex initial costs. As a result, our numerical methods have the potential to serve as a building block for solving broader classes of highdimensional optimal control problems in real-time.

[1]  Masayoshi Tomizuka,et al.  Autonomous Driving Motion Planning With Constrained Iterative LQR , 2019, IEEE Transactions on Intelligent Vehicles.

[2]  James E. Bobrow,et al.  An efficient sequential linear quadratic algorithm for solving nonlinear optimal control problems , 2005, Proceedings of the 2005, American Control Conference, 2005..

[3]  Jochen Garcke,et al.  Suboptimal Feedback Control of PDEs by Solving HJB Equations on Adaptive Sparse Grids , 2017, J. Sci. Comput..

[4]  Scott Kuindersma,et al.  An Architecture for Online Affordance‐based Perception and Whole‐body Planning , 2015, J. Field Robotics.

[5]  W.M. McEneaney,et al.  Curse-of-complexity attenuation in the curse-of-dimensionality-free method for HJB PDEs , 2008, 2008 American Control Conference.

[6]  Chi-Wang Shu,et al.  A Discontinuous Galerkin Finite Element Method for Hamilton-Jacobi Equations , 1999, SIAM J. Sci. Comput..

[7]  Maurizio Falcone,et al.  An Efficient DP Algorithm on a Tree-Structure for Finite Horizon Optimal Control Problems , 2018, SIAM J. Sci. Comput..

[8]  Samy Wu Fung,et al.  A Neural Network Approach for High-Dimensional Optimal Control , 2021, 2104.03270.

[9]  Tingwei Meng,et al.  On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton-Jacobi partial differential equations , 2021, J. Comput. Phys..

[10]  George Em Karniadakis,et al.  SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems , 2020, Neural Networks.

[11]  Frank L. Lewis,et al.  Intelligent optimal control of robotic manipulators using neural networks , 2000, Autom..

[12]  B. Kouvaritakis,et al.  Efficient MPC Optimization using Pontryagin's Minimum Principle , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[13]  William M. McEneaney,et al.  Max-plus methods for nonlinear control and estimation , 2005 .

[14]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[15]  Somil Bansal,et al.  Generating Robust Supervision for Learning-Based Visual Navigation Using Hamilton-Jacobi Reachability , 2020, L4DC.

[16]  Huan Zhang,et al.  Max-plus fundamental solution semigroups for optimal control problems , 2015, SIAM Conf. on Control and its Applications.

[17]  Masayoshi Tomizuka,et al.  Alternating Direction Method of Multipliers for Constrained Iterative LQR in Autonomous Driving , 2020, ArXiv.

[18]  Wei Zhan,et al.  Constrained iterative LQR for on-road autonomous driving motion planning , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[19]  Danping Peng,et al.  Weighted ENO Schemes for Hamilton-Jacobi Equations , 1999, SIAM J. Sci. Comput..

[20]  Stéphane Gaubert,et al.  The Max-Plus Finite Element Method for Solving Deterministic Optimal Control Problems: Basic Properties and Convergence Analysis , 2008, SIAM J. Control. Optim..

[21]  Frank L. Lewis,et al.  Robot Manipulator Control: Theory and Practice , 2003 .

[22]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[23]  Somil Bansal,et al.  DeepReach: A Deep Learning Approach to High-Dimensional Reachability , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[24]  K. G. Farlow,et al.  Max-Plus Algebra , 2009 .

[25]  J. Lygeros,et al.  Neural approximation of PDE solutions: An application to reachability computations , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[26]  Marceau Coupechoux,et al.  Optimal Trajectories of a UAV Base Station Using Lagrangian Mechanics , 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[27]  Jianfeng Lu,et al.  Actor-Critic Method for High Dimensional Static Hamilton-Jacobi-Bellman Partial Differential Equations based on Neural Networks , 2021, SIAM J. Sci. Comput..

[28]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[29]  Jérôme Darbon,et al.  On Convex Finite-Dimensional Variational Methods in Imaging Sciences and Hamilton-Jacobi Equations , 2015, SIAM J. Imaging Sci..

[30]  C. Yalçin Kaya,et al.  A Duality Approach for Solving Control-Constrained Linear-Quadratic Optimal Control Problems , 2014, SIAM J. Control. Optim..

[31]  J. Denk,et al.  Synthesis of a Walking Primitive Database for a Humanoid Robot using Optimal Control Techniques , 2001 .

[32]  Qi Gong,et al.  QRnet: Optimal Regulator Design With LQR-Augmented Neural Networks , 2020, IEEE Control Systems Letters.

[33]  H. Pham,et al.  Deep neural networks algorithms for stochastic control problems on finite horizon, part I: convergence analysis , 2020 .

[34]  Daniel Delahaye,et al.  (EN-023) Mathematical Models for Aircraft Trajectory Design : A Survey. , 2013 .

[35]  Wook Hyun Kwon,et al.  LQ tracking controls with fixed terminal states and their application to receding horizon controls , 2008, Syst. Control. Lett..

[36]  Mo Chen,et al.  Reachability-Based Safety and Goal Satisfaction of Unmanned Aerial Platoons on Air Highways , 2016, 1602.08150.

[37]  Shuai Li,et al.  Robot manipulator control using neural networks: A survey , 2018, Neurocomputing.

[38]  S. Puechmorel,et al.  On a Hamilton‐Jacobi‐Bellman approach for coordinated optimal aircraft trajectories planning , 2016 .

[39]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[40]  William M. McEneaney,et al.  A Max-Plus-Based Algorithm for a Hamilton--Jacobi--Bellman Equation of Nonlinear Filtering , 2000, SIAM J. Control. Optim..

[41]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[42]  Stefan Volkwein,et al.  Error Analysis for POD Approximations of Infinite Horizon Problems via the Dynamic Programming Approach , 2015, SIAM J. Control. Optim..

[43]  Peter M. Dower,et al.  Neural network architectures using min plus algebra for solving certain high dimensional optimal control problems and Hamilton-Jacobi PDEs , 2021, ArXiv.

[44]  Tingwei Meng,et al.  Overcoming the curse of dimensionality for some Hamilton–Jacobi partial differential equations via neural network architectures , 2019 .

[45]  William M. McEneaney,et al.  Convergence Rate for a Curse-of-Dimensionality-Free Method for a Class of HJB PDEs , 2009, SIAM J. Control. Optim..

[46]  Feng Lin,et al.  An optimal control approach to robust control of robot manipulators , 1998, IEEE Trans. Robotics Autom..

[47]  Shuuji Kajita,et al.  An optimal planning of falling motions of a humanoid robot , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[48]  Zhen Zhang,et al.  Learning Poisson systems and trajectories of autonomous systems via Poisson neural networks , 2020, ArXiv.

[49]  Claire J. Tomlin,et al.  A Hopf-Lax Formula in Hamilton–Jacobi Analysis of Reach-Avoid Problems , 2021, IEEE Control Systems Letters.

[50]  William M. McEneaney,et al.  A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs , 2007, SIAM J. Control. Optim..

[51]  Hussein Jaddu,et al.  Spectral method for constrained linear-quadratic optimal control , 2002, Math. Comput. Simul..

[52]  Wei Kang,et al.  Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equations , 2015, Computational Optimization and Applications.

[53]  Scott Kuindersma,et al.  Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot , 2015, Autonomous Robots.

[54]  Huyên Pham,et al.  Some machine learning schemes for high-dimensional nonlinear PDEs , 2019, ArXiv.

[55]  J. Lygeros,et al.  A Neural Approximation to Continuous Time Reachability Computations , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[56]  Florent Lamiraux,et al.  Optimal motion planning for humanoid robots , 2013, 2013 IEEE International Conference on Robotics and Automation.

[57]  Joel W. Burdick,et al.  Linear Hamilton Jacobi Bellman Equations in high dimensions , 2014, 53rd IEEE Conference on Decision and Control.

[58]  Alessandro Rucco,et al.  Optimal Rendezvous Trajectory for Unmanned Aerial-Ground Vehicles , 2018, IEEE Transactions on Aerospace and Electronic Systems.

[59]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[60]  Michael Muehlebach,et al.  Application of an approximate model predictive control scheme on an unmanned aerial vehicle , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Michael Griebel,et al.  An Adaptive Sparse Grid Semi-Lagrangian Scheme for First Order Hamilton-Jacobi Bellman Equations , 2012, Journal of Scientific Computing.

[62]  Emanuel Todorov,et al.  Iterative Linear Quadratic Regulator Design for Nonlinear Biological Movement Systems , 2004, ICINCO.

[63]  S. Osher,et al.  Algorithms for overcoming the curse of dimensionality for certain Hamilton–Jacobi equations arising in control theory and elsewhere , 2016, Research in the Mathematical Sciences.

[64]  M. Bardi,et al.  Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations , 1997 .

[65]  Karl Kunisch,et al.  Robust Feedback Control of Nonlinear PDEs by Numerical Approximation of High-Dimensional Hamilton-Jacobi-Isaacs Equations , 2019, SIAM J. Appl. Dyn. Syst..

[66]  D. Bertsekas.  Reinforcement Learning and Optimal ControlA Selective Overview , 2018 .

[67]  Christoph Reisinger,et al.  Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems , 2019, Analysis and Applications.

[68]  William M. McEneaney,et al.  Curse of dimensionality reduction in max-plus based approximation methods: Theoretical estimates and improved pruning algorithms , 2011, IEEE Conference on Decision and Control and European Control Conference.

[69]  Karl Kunisch,et al.  Polynomial Approximation of High-Dimensional Hamilton-Jacobi-Bellman Equations and Applications to Feedback Control of Semilinear Parabolic PDEs , 2017, SIAM J. Sci. Comput..

[70]  Claire J. Tomlin,et al.  A Computationally Efficient Hamilton-Jacobi-based Formula for State-Constrained Optimal Control Problems , 2021, ArXiv.

[71]  Tingwei Meng,et al.  On Decomposition Models in Imaging Sciences and Multi-time Hamilton-Jacobi Partial Differential Equations , 2019, SIAM J. Imaging Sci..

[72]  Lei Xie,et al.  HJB-POD-Based Feedback Design for the Optimal Control of Evolution Problems , 2004, SIAM J. Appl. Dyn. Syst..

[73]  Wang Hai-bing,et al.  High-order essentially non-oscillatory schemes for Hamilton-Jacobi equations , 2006 .

[74]  Christopher G. Atkeson,et al.  Optimization based full body control for the atlas robot , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[75]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[76]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..