Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning

Abstract The organic Rankine cycle (ORC) is a promising technology for engine waste heat recovery. During real-world operation, the engine working condition varies frequently to satisfy the power demand; thus, the transient nature of engine waste heat presents significant control challenges for the ORC. To control the superheat of the ORC precisely under a transient heat source, several optimal control methods have been used such as model predictive control and dynamic programing. However, most of them depend strongly on the accurate prediction of future disturbances. Deep reinforcement learning (DRL) is an artificial-intelligence algorithm that can overcome the aforementioned disadvantage, but the potential of DRL in control of thermodynamic systems has not yet been investigated. Thus, this paper proposes two DRL-based control methods for controlling the superheat of ORC under a transient heat source. One directly uses the DRL agent to learn the control strategy (DRL control), and the other uses the DRL agent to optimize the parameters of the proportional–integral–derivative (PID) controller (DRL-based PID control). Additionally, a switching mechanism between different DRL controllers is proposed for improving the training efficiency and enlarging the operation range of the controller. The results of this study indicate that the DRL agent can satisfactorily perform the control task and optimize the traditional controller under the trained and untrained transient heat source. Specifically, the DRL control can track the reference superheat with an average error of only 0.19 K, whereas that of the traditional PID control is 2.16 K. Furthermore, the proposed switching DRL control exhibits excellent tracking performance with an average error of only 0.21 K and robustness over a wide range of operation conditions. The successful application of DRL demonstrates its considerable potential for the control of thermodynamic systems, providing a useful reference and motivation for the application to other thermodynamic systems.

[1]  Zoran Filipi,et al.  An experimentally validated, energy focused, optimal control strategy for an Organic Rankine Cycle waste heat recovery system , 2019 .

[2]  Gequn Shu,et al.  Theoretical analysis and comparison of rankine cycle and different organic rankine cycles as waste heat recovery system for a large gaseous fuel internal combustion engine , 2016 .

[3]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[4]  Gequn Shu,et al.  Alkanes as working fluids for high-temperature exhaust heat recovery of diesel engine using organic Rankine cycle , 2014 .

[5]  Hua Tian,et al.  Part-Load Performance Prediction and Operation Strategy Design of Organic Rankine Cycles with a Medium Cycle Used for Recovering Waste Heat from Gaseous Fuel Engines , 2016 .

[6]  Jeong Ik Lee,et al.  A comprehensive design methodology of organic Rankine cycles for the waste heat recovery of automotive heavy-duty diesel engines , 2015 .

[7]  Dariusz Mikielewicz,et al.  A thermodynamic criterion for selection of working fluid for subcritical and supercritical domestic micro CHP , 2010 .

[8]  Boyuan Fan,et al.  A performance analysis of a novel system of a dual loop bottoming organic Rankine cycle (ORC) with a light-duty diesel engine , 2013 .

[9]  Weiwen Deng,et al.  Power Management for Hybrid Energy Storage System of Electric Vehicles Considering Inaccurate Terrain Information , 2017, IEEE Transactions on Automation Science and Engineering.

[10]  Andreas Kugi,et al.  Model predictive control of an automotive waste heat recovery system , 2018, Control Engineering Practice.

[11]  W. Feng,et al.  Effect factors of part-load performance for various Organic Rankine cycles using in engine waste heat recovery , 2018, Energy Conversion and Management.

[12]  Lei Xie,et al.  Fast economic nonlinear model predictive control strategy of Organic Rankine Cycle for waste heat recovery: Simulation-based studies , 2019, Energy.

[13]  Yukun Hu,et al.  Control of Supercritical Organic Rankine Cycle based Waste Heat Recovery System Using Conventional and Fuzzy Self-tuned PID Controllers , 2019, International Journal of Control, Automation and Systems.

[14]  Di Cao,et al.  Deep reinforcement learning–based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy , 2019, Energy Conversion and Management.

[15]  Maciej Wieczorek,et al.  A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm , 2017 .

[16]  Ming Jin,et al.  Control-Theoretic Analysis of Smoothness for Stability-Certified Reinforcement Learning , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[17]  Jianhua Zhang,et al.  Generalized Correntropy Predictive Control for Waste Heat Recovery Systems Based on Organic Rankine Cycle , 2019, IEEE Access.

[18]  Vincent Lemort,et al.  Dynamic modeling and optimal control strategy of waste heat recovery Organic Rankine Cycles , 2011 .

[19]  S. Tassou,et al.  An appraisal of proportional integral control strategies for small scale waste heat to power conversion units based on Organic Rankine Cycles , 2018, Energy.

[20]  Jianfei Cao,et al.  Energy optimization of electric vehicle’s acceleration process based on reinforcement learning , 2020 .

[21]  Jihie Kim,et al.  Ensemble-Based Deep Reinforcement Learning for Chatbots , 2019, Neurocomputing.

[22]  Ming Jin,et al.  Advanced Building Control via Deep Reinforcement Learning , 2019, Energy Procedia.

[23]  He Yin,et al.  Equivalent Series Resistance-Based Energy Loss Analysis of a Battery Semiactive Hybrid Energy Storage System , 2015, IEEE Transactions on Energy Conversion.

[24]  Peng Liu,et al.  Dynamic analysis of the dual-loop Organic Rankine Cycle for waste heat recovery of a natural gas engine , 2017 .

[25]  Zhibin Yu,et al.  Theoretical analysis of a regenerative supercritical carbon dioxide Brayton cycle/organic Rankine cycle dual loop for waste heat recovery of a diesel/natural gas dual-fuel engine , 2019, Energy Conversion and Management.

[26]  Peng Liu,et al.  Engine working condition effects on the dynamic response of organic Rankine cycle as exhaust waste heat recovery system , 2017 .

[27]  G. Shu,et al.  Performance assessment of engine exhaust-based segmented thermoelectric generators by length ratio optimization , 2019, Applied Energy.

[28]  A. Sanjid,et al.  Energy balance of internal combustion engines using alternative fuels , 2013 .

[29]  Z. Filipi,et al.  A comprehensive review of organic rankine cycle waste heat recovery systems in heavy-duty diesel engine applications , 2019, Renewable and Sustainable Energy Reviews.

[30]  Olivier Lepreux,et al.  Improving the control performance of an Organic Rankine Cycle system for waste heat recovery from a heavy-duty diesel engine using a model-based approach , 2013, 52nd IEEE Conference on Decision and Control.

[31]  Jingda Wu,et al.  Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle , 2019, Applied Energy.

[32]  Sergio Rech,et al.  Design and off-design models of single and two-stage ORC systems on board a LNG carrier for the search of the optimal performance and control strategy , 2017 .

[33]  Gequn Shu,et al.  Dynamic performance comparison of different cascade waste heat recovery systems for internal combustion engine in combined cooling, heating and power , 2020 .

[34]  Lucian Busoniu,et al.  Reinforcement learning for control: Performance, stability, and deep approximators , 2018, Annu. Rev. Control..

[35]  Suresh Kumar,et al.  A technical review on waste heat recovery from compression ignition engines using organic Rankine cycle , 2018 .

[36]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[37]  Jiayi Cao,et al.  Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle , 2018 .

[38]  Paolino Tona,et al.  Optimal Control for an Organic Rankine Cycle on board a Diesel-Electric Railcar , 2015 .