Agent-Based Modeling in Electricity Market Using Deep Deterministic Policy Gradient Algorithm

Game theoretic methods and simulations based on reinforcement learning (RL) are often used to analyze electricity market equilibrium. However, the former is limited to a simple market environment with complete information, and difficult to visually reflect the tacit collusion; while the conventional RL algorithm is limited to low-dimensional discrete state and action spaces, and the convergence is unstable. To address the aforementioned problems, this paper adopts deep deterministic policy gradient (DDPG) algorithm to model the bidding strategies of generation companies (GenCos). Simulation experiments, including different settings of GenCo, load and network, demonstrate that the proposed method is more accurate than conventional RL algorithm, and can converge to the Nash equilibrium of complete information even in the incomplete information environment. Moreover, the proposed method can intuitively reflect the different tacit collusion level by quantitatively adjusting GenCos’ patience parameter, which can be an effective means to analyze market strategies.

[1]  Goran Strbac,et al.  Deep Reinforcement Learning for Strategic Bidding in Electricity Markets , 2020, IEEE Transactions on Smart Grid.

[2]  Chuanwen Jiang,et al.  The Formulation of the Optimal Strategies for the Electricity Producers Based on the Particle Swarm Optimization Algorithm , 2006, IEEE Transactions on Power Systems.

[3]  Leigh Tesfatsion,et al.  Market power and efficiency in a computational electricity market with discriminatory double-auction pricing , 2001, IEEE Trans. Evol. Comput..

[4]  Guillaume J. Laurent,et al.  Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems , 2012, The Knowledge Engineering Review.

[5]  E. Bompard,et al.  Network constraint impacts on the competitive electricity markets under supply-side strategic bidding , 2006, IEEE Transactions on Power Systems.

[6]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[7]  Andrew Sweeting,et al.  Market Power in the England and Wales Wholesale Electricity Market 1995-2000 , 2004 .

[8]  Joseph E. Harrington,et al.  A Theory of Tacit Collusion , 2012 .

[9]  Rob Fergus,et al.  Modeling Others using Oneself in Multi-Agent Reinforcement Learning , 2018, ICML.

[10]  A. Roth,et al.  Learning in Extensive-Form Games: Experimental Data and Simple Dynamic Models in the Intermediate Term* , 1995 .

[11]  Zhang Yao,et al.  A Method for Designing Agent-based Electricity Market Simulation Experiments , 2009 .

[12]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[13]  Xiao-Ping Zhang.  Restructured electric power systems : analysis of electricity markets with equilibrium models , 2010 .

[14]  A.C. Tellidou,et al.  Agent-Based Analysis of Capacity Withholding and Tacit Collusion in Electricity Markets , 2007, IEEE Transactions on Power Systems.

[15]  L. Tesfatsion,et al.  An Agent-Based Test Bed Study of Wholesale Power Market Performance Measures , 2008, IEEE Computational Intelligence Magazine.

[16]  Wei Qiao,et al.  Finding Equilibria in the Pool-Based Electricity Market With Strategic Wind Power Producers and Network Constraints , 2017, IEEE Transactions on Power Systems.

[17]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[18]  Hanchen Xu,et al.  Deep Reinforcement Learning for Joint Bidding and Pricing of Load Serving Entity , 2019, IEEE Transactions on Smart Grid.

[19]  Huaguang Zhang,et al.  Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning , 2019, Energies.

[20]  Amanpreet Singh,et al.  Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks , 2018, ICLR.

[21]  R. Fletcher,et al.  Numerical experience with solving MPECs as NLPs , 2002 .

[22]  Jie Song,et al.  Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[23]  Yuzeng Li,et al.  Oligopolistic equilibrium analysis for electricity markets: a nonlinear complementarity approach , 2004, IEEE Transactions on Power Systems.

[24]  J. Price,et al.  Evaluation of Market Rules Using a Multi-Agent System Method , 2010, IEEE Transactions on Power Systems.

[25]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[26]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[27]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[28]  J. Friedman.  A Non-cooperative Equilibrium for Supergames , 1971 .

[29]  Renke Huang,et al.  Adaptive Power System Emergency Control Using Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[30]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[31]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[32]  Haibo He,et al.  Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[33]  Robert Gibbons,et al.  A primer in game theory , 1992 .

[34]  Chen Qingsong,et al.  Realization of Decision-making Module in Agent-based Simulation of Power Markets , 2008 .

[35]  Albert Banal-Estañol,et al.  Behavioural simulations in spot electricity markets , 2011, Eur. J. Oper. Res..

[36]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.