A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines

Nowadays, liquid rocket engines use closed-loop control at most near steady operating conditions. The control of the transient phases is traditionally performed in open-loop due to highly nonlinear system dynamics. This situation is unsatisfactory, in particular for reusable engines. The open-loop control system cannot provide optimal engine performance due to external disturbances or the degeneration of engine components over time. In this paper, we study a deep reinforcement learning approach for optimal control of a generic gas-generator engine's continuous start-up phase. It is shown that the learned policy can reach different steady-state operating points and convincingly adapt to changing system parameters. A quantitative comparison with carefully tuned open-loop sequences and PID controllers is included. The deep reinforcement learning controller achieves the highest performance and requires only minimal computational effort to calculate the control action, which is a big advantage over approaches that require online optimization, such as model predictive control. control.

[1]  N. Girard,et al.  PROMETHEUS , A LOX / LCH 4 REUSABLE ROCKET ENGINE , 2017 .

[2]  Julien Marzat,et al.  A survey of automatic control methods for liquid-propellant rocket engines , 2019, Progress in Aerospace Sciences.

[3]  Javad Lavaei,et al.  Stability-Certified Reinforcement Learning: A Control-Theoretic Perspective , 2018, IEEE Access.

[4]  Julien Marzat,et al.  Derivation and Analysis of a State-Space Model for Transient Control of Liquid-Propellant Rocket Engines , 2018, 2018 9th International Conference on Mechanical and Aerospace Engineering (ICMAE).

[5]  Hussein A. Abbass,et al.  Hierarchical Deep Reinforcement Learning for Continuous Action Control , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Julien Marzat,et al.  An MPC Approach to Transient Control of Liquid-Propellant Rocket Engines , 2019, IFAC-PapersOnLine.

[7]  Roberto Furfaro,et al.  Deep reinforcement learning for six degree-of-freedom planetary landing , 2020 .

[8]  Takuya Akiba,et al.  Optuna: A Next-generation Hyperparameter Optimization Framework , 2019, KDD.

[9]  Francois Lassoudiere,et al.  Progress of the development of an all-electric control system of a rocket engine , 2012 .

[10]  Jeffrey L. Musgrave,et al.  Overview of rocket engine control , 2008 .

[11]  Frank L. Lewis,et al.  Optimal and Autonomous Control Using Reinforcement Learning: A Survey , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Ed Nemeth,et al.  Reusable rocket engine intelligent control system framework design, phase 2 , 1991 .

[13]  Sergey Levine,et al.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Sergey Levine,et al.  Reinforcement Learning with Deep Energy-Based Policies , 2017, ICML.

[15]  D. Bertsekas.  Reinforcement Learning and Optimal ControlA Selective Overview , 2018 .

[16]  Frank L. Lewis,et al.  Dynamic Multiobjective Control for Continuous-Time Systems Using Reinforcement Learning , 2019, IEEE Transactions on Automatic Control.

[17]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[18]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  R. B. Gopaluni,et al.  Deep reinforcement learning approaches for process control , 2017, 2017 6th International Symposium on Advanced Control of Industrial Processes (AdCONIP).

[20]  Jan Deeken,et al.  Numerically Efficient Fatigue Life Prediction of Rocket Combustion Chambers using Artificial Neural Networks , 2019 .

[21]  Marc Parizeau,et al.  DEAP: evolutionary algorithms made easy , 2012, J. Mach. Learn. Res..

[22]  Amir Hussain,et al.  Applications of Deep Learning and Reinforcement Learning to Biological Data , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[24]  Lucian Busoniu,et al.  Reinforcement learning for control: Performance, stability, and deep approximators , 2018, Annu. Rev. Control..

[25]  Erik-Jan Van Kampen,et al.  Online Adaptive Incremental Reinforcement Learning Flight Control for a CS-25 Class Aircraft , 2020, AIAA Scitech 2020 Forum.

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  L. Meland,et al.  History of the Titan liquid rocket engines , 1989 .

[28]  Antoine Iffly,et al.  Performance model of the Vulcain Ariane 5 main engine , 1999 .

[29]  Günther Waxenegger-Wilfing,et al.  Heat Transfer Prediction for Methane in Regenerative Cooling Channels with Neural Networks , 2019, Journal of Thermophysics and Heat Transfer.

[30]  Thomas Frohlich,et al.  LOX/METHANE AND LOX/KEROSENE HIGH THRUST ENGINE TRADE-OFF , 2001 .

[31]  Michael E. Hampson.  Reusable Rocket Engine Turbopump Condition Monitoring , 1984 .

[32]  Pieter Abbeel,et al.  Constrained Policy Optimization , 2017, ICML.

[33]  Xuesong Wang,et al.  Approximate Policy-Based Accelerated Deep Reinforcement Learning , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[34]  R. Mazo.  On the theory of brownian motion , 1973 .

[35]  N. Munro,et al.  PID controllers: recent tuning methods and design to specification , 2002 .

[36]  Daniel E. Paxson,et al.  A demonstration of an intelligent control system for a reusable rocket engine , 1992 .

[37]  Herke van Hoof,et al.  Addressing Function Approximation Error in Actor-Critic Methods , 2018, ICML.

[38]  R. M. Ryan,et al.  Effects of geometry and materials on low cycle fatigue life of turbine blades in LOX/hydrogen rocket engines , 1986 .

[39]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[40]  R. J. Roelke.  Miscellaneous losses. [tip clearance and disk friction] , 1973 .