Model-based deep reinforcement learning for wind energy bidding

Abstract Wind energy is an important source of clean energy. Due to the common trade through bidding, many attempts have been made to apply deep reinforcement learning techniques to generate appropriate bidding policies to maximize profits. However, these algorithms are based entirely on a model-free strategy. The present study aims to develop a dynamic model capable of strategic bidding for wind energy. Thus, the model MB-A3C is implemented and proves to be quite resilient. Herein, “Nord Pool”, a conventional benchmark that comprises six datasets representing each wind power site in Denmark and Sweden is duly investigated. Results show that the policies generated by MB-A3C are less costly than those produced by both previous model-free and model-based algorithms i.e. Conv-A3C, DPPO, DDPG, and MBPG. The optimal bidding approach demonstrated in this study can be utilized to optimize profits and overcome the uncertainties in both the energy and reserve markets.

[1]  Depeng Wang,et al.  Short-Term Electricity Demand Forecasting Using ComponentsEstimation Technique , 2019, Energies.

[2]  Tomoaki Ohtsuki,et al.  Deep Reinforcement Learning for Economic Dispatch of Virtual Power Plant in Internet of Energy , 2020, IEEE Internet of Things Journal.

[3]  Ke Yan,et al.  Short-Term Solar Irradiance Forecasting Based on a Hybrid Deep Learning Methodology , 2020, Inf..

[4]  Ran Jing,et al.  A Self-attention Based LSTM Network for Text Classification , 2019, Journal of Physics: Conference Series.

[5]  Sergey Levine,et al.  Temporal Difference Models: Model-Free Deep RL for Model-Based Control , 2018, ICLR.

[6]  Bowon Lee,et al.  Automatic P2P Energy Trading Model Based on Reinforcement Learning Using Long Short-Term Delayed Reward , 2020, Energies.

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  Francesco Lisi,et al.  Forecasting next-day electricity demand and prices based on functional models , 2019, Energy Systems.

[9]  Marco Wiering,et al.  Reinforcement Learning and Markov Decision Processes , 2012, Reinforcement Learning.

[10]  Ali Etemad,et al.  Classification of Hand Movements From EEG Using a Deep Attention-Based LSTM Network , 2019, IEEE Sensors Journal.

[11]  Judith Foster,et al.  Load forecasting techniques for power systems with high levels of unmetered renewable generation: A comparative study , 2018 .

[12]  Julien Jacques,et al.  Short-Term Electricity Demand Forecasting Using a Functional State Space Model , 2018 .

[13]  Richard S. Sutton,et al.  Dyna, an integrated architecture for learning, planning, and reacting , 1990, SGAR.

[14]  Qingyu Yang,et al.  Energy Trading in Smart Grid: A Deep Reinforcement Learning-based Approach , 2020, 2020 Chinese Control And Decision Conference (CCDC).

[15]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[16]  Xin Wang,et al.  Model-based Policy Gradient Reinforcement Learning , 2003, ICML.

[17]  Pengbo Li,et al.  Short‐term wind power forecasting based on two‐stage attention mechanism , 2020, IET Renewable Power Generation.

[18]  R. Weron,et al.  Recent advances in electricity price forecasting: A review of probabilistic forecasting , 2016 .

[19]  Xinying Wang,et al.  Deep Reinforcement Learning Approach for Autonomous Agents in Consumer-centric Electricity Market , 2020, 2020 5th IEEE International Conference on Big Data Analytics (ICBDA).

[20]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[21]  Easwar Subramanian,et al.  Bidding Strategy for Two-Sided Electricity Markets: A Reinforcement Learning based Framework , 2020, BuildSys@SenSys.

[22]  Wei Gu,et al.  Combined heat and power system intelligent economic dispatch: A deep reinforcement learning approach , 2020, International Journal of Electrical Power & Energy Systems.

[23]  Brian D. Deaton.  Effects of the Swiss Franc/Euro Exchange Rate Floor on the Calibration of Probability Forecasts , 2018 .

[24]  Yuval Tassa,et al.  Emergence of Locomotion Behaviours in Rich Environments , 2017, ArXiv.

[25]  Mitch Campion,et al.  A Multi-Stage Price Forecasting Model for Day-Ahead Electricity Markets , 2018, Forecasting.

[26]  Lei Shi,et al.  Subsidy-Free Renewable Energy Trading: A Meta Agent Approach , 2020, IEEE Transactions on Sustainable Energy.

[27]  Mirza Mulaosmanovic,et al.  SHORT-TERM ELECTRICITY PRICE FORECASTING ON THE NORD POOL MARKET , 2017 .

[28]  Ioannis P. Panapakidis,et al.  Day-ahead electricity price forecasting via the application of artificial neural network based models , 2016 .

[29]  Lars Herre,et al.  Exploring wind power prognosis data on Nord Pool: the case of Sweden and Denmark , 2019, IET Renewable Power Generation.

[30]  R. Weron.  Electricity price forecasting: A review of the state-of-the-art with a look into the future , 2014 .

[31]  Zhe Chen,et al.  A data-driven approach for designing STATCOM additional damping controller for wind farms , 2020 .

[32]  Mengxia Wang,et al.  A deep reinforcement learning method for managing wind farm uncertainties through energy storage system control and external reserve purchasing , 2020 .

[33]  T. Kristiansen.  Forecasting Nord Pool day-ahead prices with an autoregressive model , 2012 .

[34]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[35]  T. Dragičević,et al.  Bidding strategy for trading wind energy and purchasing reserve of wind power producer – A DRL based approach , 2020 .

[36]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[37]  George Xydis,et al.  Electricity Price Forecasting in the Danish Day-Ahead Market Using the TBATS, ANN and ARIMA Methods , 2019, Energies.

[38]  Limin Wang,et al.  A Deep Reinforcement Learning Bidding Algorithm on Electricity Market , 2020, Journal of Thermal Science.

[39]  G. Xydis,et al.  High-Resolution Electricity Spot Price Forecast for the Danish Power Market , 2020, Sustainability.

[40]  Johannes Krokeide Kolberg,et al.  Artificial Intelligence and Nord Pool’s intraday electricity market Elbas : a demonstration and pragmatic evaluation of employing deep learning for price prediction : using extensive market data and spatio-temporal weather forecasts , 2018 .

[41]  Sergey Levine,et al.  When to Trust Your Model: Model-Based Policy Optimization , 2019, NeurIPS.

[42]  Jianguo Yao,et al.  A parallel multi-scenario learning method for near-real-time power dispatch optimization , 2020 .

[43]  Yuandong Tian,et al.  Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees , 2018, ICLR.