Deep reinforcement learning–based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy

Abstract With advanced information technologies applied in integrated energy systems (IESs), controlling the energy conversion has become an effective method for improving grid flexibility and reducing the operating cost of IESs. This study proposes a dynamic energy conversion strategy for the energy management of an IES with renewable energy, which considers the system operator’s (SO) operating cost. Deep reinforcement learning (DRL) is used to illustrate the hierarchical decision-making process, in which the dynamic energy conversion problem is formulated as a discrete finite Markov decision process, and proximal policy optimization (PPO) is adopted to solve the decision-making problem. Using DRL, the SO can adaptively decide the wind power conversion ratio during the online learning process, where the uncertainties of customers’ load demand profiles, flexibility of spot electricity prices, and wind power generation are addressed. Simulations show that the proposed PPO-based renewable energy conversion algorithm can effectively reduce the SO’s operating cost.

[1]  J. H. Harker,et al.  Fluid flow, heat transfer and mass transfer , 1996 .

[2]  Seyed Ali Ghoreishi-Madiseh,et al.  Study of effective solar energy storage using a double pipe geothermal heat exchanger , 2016 .

[3]  Wen‐ying Li,et al.  Coke oven gas to methanol process integrated with CO2 recycle for high energy efficiency, economic benefits and low emissions , 2017 .

[4]  Ibrahim Dincer,et al.  Investigation and assessment of a novel solar-driven integrated energy system , 2018 .

[5]  Haibo He,et al.  Optimal Real-Time Operation Strategy for Microgrid: An ADP-Based Stochastic Nonlinear Optimization Approach , 2019, IEEE Transactions on Sustainable Energy.

[6]  Hemanshu R. Pota,et al.  Distributed multi-agent scheme for reactive power management with renewable energy , 2014 .

[7]  Mohamed Elsholkami,et al.  New technology integration approach for energy planning with carbon emission considerations , 2015 .

[8]  Kenneth Bernard Karlsson,et al.  Residential heat pumps in the future Danish energy system , 2016 .

[9]  Risto Lahdelma,et al.  Modelling and optimization of CHP based district heating system with renewable energy production and energy storage , 2015 .

[10]  Guangya Zhu,et al.  Design optimisation and performance appraisal of a combined cooling, heating and power system primed with Maisotsenko combustion turbine cycle , 2018, Energy Conversion and Management.

[11]  Ibrahim Dincer,et al.  Development, analysis and assessment of solar energy-based multigeneration system with thermoelectric generator , 2018 .

[12]  Rajagopal Peesapati,et al.  Flower pollination algorithm based multi-objective congestion management considering optimal capacities of distributed generations , 2018 .

[13]  Mohsen Guizani,et al.  Semisupervised Deep Reinforcement Learning in Support of IoT and Smart City Services , 2018, IEEE Internet of Things Journal.

[14]  Sonja Wogrin,et al.  Generation flexibility in ramp rates: Strategic behavior and lessons for electricity market design , 2017, Eur. J. Oper. Res..

[15]  Dongdong Zhang,et al.  Standardized modelling and economic optimization of multi-carrier energy systems considering energy storage and demand response , 2019, Energy Conversion and Management.

[16]  Xiaobo Liu,et al.  Optimization analysis of a novel combined heating and power system based on biomass partial gasification and ground source heat pump , 2018 .

[17]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[18]  Abdelali Astito,et al.  Sizing a PV-Wind based hybrid system using deterministic approach , 2018, Energy Conversion and Management.

[19]  J. McCalley,et al.  A Multiperiod Generalized Network Flow Model of the U.S. Integrated Energy System: Part I—Model Description , 2007, IEEE Transactions on Power Systems.

[20]  Muyiwa S. Adaramola,et al.  Assessment of Wind Power Generation Along the Coast of Ghana , 2014 .

[21]  Bo Tang,et al.  A Generative Model for Sparse Hyperparameter Determination , 2018, IEEE Transactions on Big Data.

[22]  Lin Lu,et al.  Optimal design and analysis of a new CHP-HP integrated system , 2017 .

[23]  Barbara Mayer,et al.  Management of hybrid energy supply systems in buildings using mixed-integer model predictive control , 2015 .

[24]  Junwei Cao,et al.  Optimal energy management strategies for energy Internet via deep reinforcement learning approach , 2019, Applied Energy.

[25]  Shilei Lu,et al.  Study on the configuration and operation optimization of CCHP coupling multiple energy system , 2018, Energy Conversion and Management.

[26]  Oliver Schütze,et al.  A Neural Network-Evolutionary Computational Framework for Remaining Useful Life Estimation of Mechanical Systems , 2019, Neural Networks.

[27]  Lingfeng Wang,et al.  Stochastic combined heat and power dispatch based on multi-objective particle swarm optimization , 2006, 2006 IEEE Power Engineering Society General Meeting.

[28]  Ursula Eicker,et al.  Impact of load structure variation and solar thermal energy integration on an existing district heating network , 2013 .

[29]  Daniele Testi,et al.  Optimal integrated sizing and operation of a CHP system with Monte Carlo risk analysis for long-term uncertainty in energy demands , 2018 .

[30]  Zhiyong Tang,et al.  Performance analysis of the wind energy integrated with a natural-gas-to-methanol process , 2018, Energy Conversion and Management.

[31]  Woojin Cho,et al.  A simple sizing method for combined heat and power units , 2014 .

[32]  Barack Obama,et al.  The irreversible momentum of clean energy , 2017, Science.

[33]  Mohamed Haouari,et al.  Review of optimization techniques applied for the integration of distributed generation from renewable energy sources , 2017 .

[34]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[35]  E. Michaelides,et al.  Substitution of coal power plants with renewable energy sources – Shift of the power demand and energy storage , 2018 .

[36]  Min Chen,et al.  An adaptive deep Q-learning strategy for handwritten digit recognition , 2018, Neural Networks.

[37]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[38]  Lauri Kujanpää,et al.  Techno-economic Evaluation of Flexible CCS Concepts in a CHP System , 2017 .

[39]  A. H. Mosaffa,et al.  Thermoeconomic assessment of a novel integrated biomass based power generation system including gas turbine cycle, solid oxide fuel cell and Rankine cycle , 2018 .

[40]  Willett Kempton,et al.  The challenge of integrating offshore wind power in the U.S. electric grid. Part II: Simulation of electricity market operations , 2017 .

[41]  Jinyu Wen,et al.  Stochastic Optimization of Economic Dispatch for Microgrid Based on Approximate Dynamic Programming , 2019, IEEE Transactions on Smart Grid.

[42]  Zishuo Huang,et al.  Energetic and exergetic analysis of integrated energy system based on parametric method , 2017 .

[43]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.