A novel energy management method based on Deep Q Network algorithm for low operating cost of an integrated hybrid system

Abstract For minimizing the operational cost of an integrated hybrid system, which provides electricity, heating, cooling and water of a residential complex, a new energy management method presented here. The proposed energy management method uses the principles of Reinforcement Learning (RL) based on Deep Q Network (DQN) algorithm to find the operational strategy for the hybrid system. Unlike the stochastic presentation of RL with Markov Decision Process (MDP) in literature, a deterministic approach proposed in this study for energy management of a hybrid system. This presented approach does not require online trial and error or historical data of system performance to determine the operational strategy. This developed RL-DQN method used for energy management of a hybrid system with photovoltaic collectors, ET collectors, wind turbines, CHP units, a water heater, a hot water storage tank and batteries. Thus for a residential complex with variable electrical and thermal loads during a year (8760 h), RL-DQN energy management method estimated two unknown variables in operational strategy, i.e., the partial load of the CHP unit and the amount of heat generation by LPG-fueled water heater. The results of RL-DQN energy management also compared with results of two rule-based methods. The results show a reduction of the operating cost to 10983 $/year by RL-DQN which is lower 6.5% and 30.4% than that for two above mentioned rule-based methods. Lower CO2 production (2.8% and 62.8%) also observed in comparison with that for two above mentioned rule based methods. With the proposed EM method, 80.1% of the required electricity supplied with renewable energies and 82.2% of the thermal load supplied without direct consumption of fossil fuels. The benefit of using predicted load and renewable energies of future hours also investigated for EM. Results show that employing the predicted information decrease the operational cost by about 3.5 percent

[1]  Jyotirmay Mathur,et al.  Implications of short-term renewable energy resource intermittency in long-term power system planning , 2018, Energy Strategy Reviews.

[2]  Youakim Badr,et al.  Energy-efficient heating control for smart buildings with deep reinforcement learning , 2020 .

[3]  Hongwen He,et al.  Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus , 2019, Applied Energy.

[4]  Hamidreza Zareipour,et al.  Residential energy management using a moving window algorithm , 2012, 2012 3rd IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe).

[5]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[6]  Enrique López Droguett,et al.  Operation scheduling in a solar thermal system: A reinforcement learning-based framework , 2020 .

[7]  Peter Henderson,et al.  An Introduction to Deep Reinforcement Learning , 2018, Found. Trends Mach. Learn..

[8]  Saad Mekhilef,et al.  Energy Management and Optimization of a PV/Diesel/Battery Hybrid Energy System Using a Combined Dispatch Strategy , 2019, Sustainability.

[9]  Nikita Tomin,et al.  Deep Reinforcement Learning for Energy Microgrids Management Considering Flexible Energy Sources , 2019, EPJ Web of Conferences.

[10]  Mario Liebensteiner,et al.  Do Intermittent Renewables Threaten the Electricity Supply Security? , 2020 .

[11]  Enrico Zambolin,et al.  An improved procedure for the experimental characterization of optical efficiency in evacuated tube solar collectors , 2012 .

[12]  Huaguang Zhang,et al.  Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning , 2019, Energies.

[13]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[14]  Erkan Dursun,et al.  Comparative evaluation of different power management strategies of a stand-alone PV/Wind/PEMFC hybrid power system , 2012 .

[15]  Hongguang Jin,et al.  A review on the utilization of hybrid renewable energy , 2018, Renewable and Sustainable Energy Reviews.

[16]  Jamshid Aghaei,et al.  Multi-objective self-scheduling of CHP (combined heat and power)-based microgrids considering demand response programs and ESSs (energy storage systems) , 2013 .

[17]  Sepehr Sanaye,et al.  Optimization of combined cooling, heating and power generation by a solar system , 2015 .

[18]  C. Byron Winn,et al.  Optimal dispatch strategy in remote hybrid power systems , 1996 .

[19]  Jay Burch,et al.  Development of Standardized Domestic Hot Water Event Schedules for Residential Buildings , 2008 .

[20]  Hyuk Lim,et al.  Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings , 2018, Energies.

[21]  Dirk Müller,et al.  Application of two promising Reinforcement Learning algorithms for load shifting in a cooling supply system , 2020, Energy and Buildings.

[22]  Santiago Grijalva,et al.  A Review of Reinforcement Learning for Autonomous Building Energy Management , 2019, Comput. Electr. Eng..

[23]  James F. Manwell,et al.  Book Review: Wind Energy Explained: Theory, Design and Application , 2006 .

[24]  Mikhail Sorin,et al.  Explicit model of photovoltaic panels to determine voltages and currents at the maximum power point , 2011 .

[25]  José L. Bernal-Agustín,et al.  Comparison of different lead–acid battery lifetime prediction models for use in simulation of stand-alone photovoltaic systems , 2014 .

[26]  M. Eastment,et al.  Development of an Energy Savings Benchmark for All Residential End-Uses: Preprint , 2004 .

[27]  R. P. Saini,et al.  Dispatch strategies based performance analysis of a hybrid renewable energy system for a remote rural area in India , 2020 .

[28]  Qi Huang,et al.  Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: Deep reinforcement learning approach , 2020 .

[29]  Sohrab Asgarpoor,et al.  Reinforcement Learning Approach for Optimal Distributed Energy Management in a Microgrid , 2018, IEEE Transactions on Power Systems.

[30]  Shye-Chorng Kuo,et al.  Comparative analysis on power curve models of wind turbine generator in estimating capacity factor , 2014 .

[31]  Ofira Ayalon,et al.  Harnessing households to mitigate renewables intermittency in the smart grid , 2019, Renewable Energy.

[32]  Jagdeep Singh,et al.  HYBRID RENEWABLE ENERGY SYSTEM: A REVIEW , 2019 .

[33]  H. Hottel,et al.  Evaluation of flat-plate solar collector performance , 1955 .

[34]  Mevludin Glavic,et al.  (Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives , 2019, Annu. Rev. Control..

[35]  Wenfeng Gao,et al.  Optimal tilt-angles of all-glass evacuated tube solar collectors , 2009 .

[36]  Sarat Kumar Sahoo,et al.  A review on state of art development of model predictive control for renewable energy applications , 2017 .

[37]  Hongqiang Guo,et al.  A robust co-state predictive model for energy management of plug-in hybrid electric bus , 2020 .

[38]  Jianxue Chai,et al.  Evaluating the comprehensive benefit of hybrid energy system for ecological civilization construction in China , 2021 .

[39]  Berhane H. Gebreslassie,et al.  Solar assisted absorption cooling cycles for reduction of global warming: A multi-objective optimization approach , 2012 .

[40]  Saad Mekhilef,et al.  Energy management and performance evaluation of grid connected PV-battery hybrid system with inherent control scheme , 2018 .

[41]  Peter Vrancx,et al.  Reinforcement Learning: State-of-the-Art , 2012 .

[42]  M. Jahangir,et al.  Feasibility study of a zero emission PV/Wind turbine/Wave energy converter hybrid system for stand-alone power supply: A case study , 2020 .

[43]  Luis M. Fernández,et al.  Sizing optimization, dynamic modeling and energy management strategies of a stand-alone PV/hydrogen/battery-based hybrid system , 2013 .

[44]  Enrico Zio,et al.  Reinforcement learning for microgrid energy management , 2013 .

[45]  Ming-Lang Tseng,et al.  Hybrid power systems with emission minimization: Multi-objective optimal operation , 2020 .

[46]  Pekka Toivanen,et al.  Deep Reinforcement Learning for Energy Management in a Microgrid with Flexible Demand , 2021 .

[47]  Saad Mekhilef,et al.  Mix-mode energy management strategy and battery sizing for economic operation of grid-tied microgrid , 2017 .