Bidding Strategy for Two-Sided Electricity Markets: A Reinforcement Learning based Framework

We aim to increase the revenue or reduce the purchase cost of a given market participant in a double-sided, day-ahead, wholesale electricity market serving a smart city. Using an operations research based market clearing mechanism and attention based time series forecaster as sub-modules, we build a holistic interactive system. Through this system, we discover better bidding strategies for a market participant using reinforcement learning (RL). We relax several assumptions made in existing literature in order to make the problem setting more relevant to real life. Our Markov Decision Process (MDP) formulation enables us to tackle action space explosion and also compute optimal actions across time-steps in parallel. Our RL framework is generic enough to be used by either a generator or a consumer participating in the electricity market. We study the efficacy of the proposed RL based bidding framework from the perspective of a generator as well as a buyer on real world day-ahead electricity market data obtained from the European Power Exchange (EPEX). We compare the performance of our RL based bidding framework against three baselines: (a) an ideal but un-realizable bidding strategy; (b) a realizable approximate version of the ideal strategy; and (c) historical performance as found from the logs. Under both perspectives, we find that our RL based framework is more closer to the ideal strategy than other baselines. Further, the RL based framework improves the average daily revenue of the generator by nearly €7,200 (€2.64 M per year) and €9,000 (€3.28 M per year) over the realizable ideal and historical strategies respectively. When used on behalf of a buyer, it reduces average daily procurement cost by nearly €2,700 (€0.97 M per year) and €57,200 (€52.63 M per year) over the realizable ideal and historical strategies respectively. We also observe that our RL based framework automatically adapts its actions to changes in the market power of the participant.

[1]  Goran Strbac,et al.  Deep Reinforcement Learning for Strategic Bidding in Electricity Markets , 2020, IEEE Transactions on Smart Grid.

[2]  H. Young,et al.  Handbook of Game Theory with Economic Applications , 2015 .

[3]  Wolfgang Ketter,et al.  The 2018 Power Trading Agent Competition , 2017 .

[4]  Sanjay P. Bhat,et al.  lEarn: A Reinforcement Learning Based Bidding Strategy for Generators in Single sided Energy Markets , 2019, e-Energy.

[5]  Filipe Joel Soares,et al.  Optimal bidding strategy for an aggregator of prosumers in energy and secondary reserve markets , 2019, Applied Energy.

[6]  S. Chatterjee,et al.  A comprehensive review on day‐ahead electricity market and important features of world's major electric power exchanges , 2020 .

[7]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[8]  Xiaofeng Wang,et al.  Reinforcement Learning to Play an Optimal Nash Equilibrium in Team Markov Games , 2002, NIPS.

[9]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[10]  Dhananjay K. Gode,et al.  Allocative Efficiency of Markets with Zero-Intelligence Traders: Market as a Partial Substitute for Individual Rationality , 1993, Journal of Political Economy.

[11]  F. J. Soares,et al.  Trading Small Prosumers Flexibility in the Energy and Tertiary Reserve Markets , 2019, IEEE Transactions on Smart Grid.

[12]  Robert B. Wilson.  Chapter 8 Strategic analysis of auctions , 1992 .

[13]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[14]  Hamidreza Zareipour,et al.  Developing Bidding and Offering Curves of a Price-Maker Energy Storage Facility Based on Robust Optimization , 2019, IEEE Transactions on Smart Grid.

[15]  Sylvie Thiébaux,et al.  Network-aware Participation of Aggregators in NEM Energy and FCAS Markets , 2020, e-Energy.

[16]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[17]  Christopher Kiekintveld,et al.  Bidding in Periodic Double Auctions Using Heuristics and Dynamic Monte Carlo Tree Search , 2018, IJCAI.

[18]  Michael P. Wellman,et al.  Flexible double auctions for electronic commerce: theory and implementation , 1998, Decis. Support Syst..

[19]  K. Neuhoff,et al.  Bidding Structures and Trading Arrangements for Flexibility across EU Power Markets , 2015 .

[20]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[21]  Gerald Tesauro,et al.  Strategic sequential bidding in auctions using dynamic programming , 2002, AAMAS '02.

[22]  B. Speer,et al.  Role of Smart Grids in Integrating Renewable Energy , 2015 .

[23]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[24]  Anand Sivasubramaniam,et al.  Windy with a chance of profit: bid strategy and analysis for wind integration , 2014, e-Energy.

[25]  Damien Ernst,et al.  Nash Equilibria and Reinforcement Learning for Active Decision Maker Modelling in Power Markets , 2004 .

[26]  Mohammad Bagher Naghibi-Sistani,et al.  Application of Q-learning with temperature variation for bidding strategies in market based power systems , 2006 .

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.