A Reinforcement Learning Model to Assess Market Power Under Auction-Based Energy Pricing

Auctions serve as a primary pricing mechanism in various market segments of a deregulated power industry. In day-ahead (DA) energy markets, strategies such as uniform price, discriminatory, and second-price uniform auctions result in different price settlements and thus offer different levels of market power. In this paper, we present a nonzero sum stochastic game theoretic model and a reinforcement learning (RL)-based solution framework that allow assessment of market power in DA markets. Since there are no available methods to obtain exact analytical solutions of stochastic games, an RL-based approach is utilized, which offers a computationally viable tool to obtain approximate solutions. These solutions provide effective bidding strategies for the DA market participants. The market powers associated with the bidding strategies are calculated using well-known indexes like Herfindahl-Hirschmann index and Lerner index and two new indices, quantity modulated price index (QMPI) and revenue-based market power index (RMPI), which are developed in this paper. The proposed RL-based methodology is tested on a sample network

[1]  Vivek S. Borkar,et al.  Learning Algorithms for Markov Decision Processes with Average Cost , 2001, SIAM J. Control. Optim..

[2]  Nils-Henrik M. von der Fehr,et al.  Competition in Electricity Spot Markets. Economic Theory and International Experience. , 1998 .

[3]  R. Baldick,et al.  Solving three-player games by the matrix approach with application to an electric power market , 2003 .

[4]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[5]  S. Oren,et al.  The efficiency of multi-unit electricity auctions , 1999 .

[6]  Fushuan Wen,et al.  Market power in generation markets , 2000 .

[7]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[8]  R. Baldick,et al.  Tuning of Discretization in Bimatrix Game Approach to Power System Market Analysis , 2002, IEEE Power Engineering Review.

[9]  S. Borenstein,et al.  An Empirical Analysis of the Potential for Market Power in California&Apos;S Electricity Industry , 1998 .

[10]  Roy Jensen,et al.  Power System Economics , 1983 .

[11]  John Moody,et al.  Learning rate schedules for faster stochastic gradient search , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[12]  Jun Li.  Learning Average Reward Irreducible Stochastic Games: Analysis and Applications , 2003 .

[13]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[14]  Computing Cournot equilibria in two settlement electricity markets with transmission constraint , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[15]  S. Stoft.  Power System Economics: Designing Markets for Electricity , 2002 .

[16]  Derek W. Bunn,et al.  Evaluating Individual Market Power in Electricity Markets via Agent-Based Simulation , 2003, Ann. Oper. Res..

[17]  Tapas K. Das,et al.  A reinforcement learning approach to a single leg airline revenue management problem with multiple fare classes and overbooking , 2002 .

[18]  S. Mahadevan,et al.  Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning , 1999 .

[19]  B. Hobbs,et al.  Linear Complementarity Models of Nash-Cournot Competition in Bilateral and POOLCO Power Markets , 2001, IEEE Power Engineering Review.

[20]  Yves Smeers,et al.  Alternative Models of Restructured Electricity Systems, Part 1: No Market Power , 2001, Oper. Res..

[21]  Yves Smeers,et al.  Spatial Oligopolistic Electricity Models with Cournot Generators and Regulated Transmission Prices , 1999, Oper. Res..

[22]  S. Mahadevan,et al.  Hierarchical Reinforcement Learning Using Graphical Models , 2022 .

[23]  Natalia Fabra,et al.  Designing Electricity Auctions: Uniform, Discriminatory and Vickrey ¤ , 2003 .

[24]  A. Rudkevich,et al.  Modeling Electricity Pricing in a Deregulated Generation Industry: The Potential for Oligopoly Pricing in a Poolco , 1998 .

[25]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[26]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[27]  Benjamin F. Hobbs,et al.  Equilibrium market power modeling for large scale power systems , 2001, 2001 Power Engineering Society Summer Meeting. Conference Proceedings (Cat. No.01CH37262).

[28]  R. Baldick,et al.  Short-term electricity market auction game analysis: uniform and pay-as-bid pricing , 2004, IEEE Transactions on Power Systems.

[29]  F.D. Galiana,et al.  Pay-as-bid versus marginal pricing-part II: market behavior under strategic Generator offers , 2004, IEEE Transactions on Power Systems.

[30]  Leigh Tesfatsion,et al.  Market power and efficiency in a computational electricity market with discriminatory double-auction pricing , 2001, IEEE Trans. Evol. Comput..

[31]  Steven Stoft,et al.  Market power in California electricity markets , 1995 .

[32]  Market Size, Market Structure & Market Power in the Irish Electricity Industry , 2005 .

[33]  B. Hobbs,et al.  Chapter 3 Complementarity-Based Equilibrium Modeling for Electric Power Markets , 2003 .

[34]  Stephen Spear.  The electricity market game , 2003, J. Econ. Theory.

[35]  J. Contreras,et al.  Auction Design in Day-Ahead Electricity Markets , 2001, IEEE Power Engineering Review.

[36]  J. Pang,et al.  Strategic gaming analysis for electric power systems: an MPEC approach , 2000 .

[37]  P. Klemperer,et al.  Supply Function Equilibria in Oligopoly under Uncertainty , 1989 .

[38]  Benjamin F. Hobbs,et al.  Understanding how market power can arise in network competition: a game theoretic approach , 1999 .

[39]  Kiran Kumar Ravulapati,et al.  A reinforcement learning approach to stochastic business games , 2004 .

[40]  J. Filar,et al.  Competitive Markov Decision Processes , 1996 .

[41]  Peter Cramton,et al.  Electricity market design: the good, the bad, and the ugly , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[42]  Yves Smeers,et al.  Variational Inequality Models of Restructured Electricity Systems , 2001 .

[43]  Michael P. Wellman,et al.  Nash Q-Learning for General-Sum Stochastic Games , 2003, J. Mach. Learn. Res..

[44]  S. A. Soman,et al.  Application of Actor-Critic Learning Algorithm for Optimal Bidding Problem of a Genco , 2002, IEEE Power Engineering Review.

[45]  Abhijit Gosavi,et al.  Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning , 2003 .

[46]  T. Das,et al.  A stochastic game approach for modeling wholesale energy bidding in deregulated power markets , 2004, IEEE Transactions on Power Systems.