Using Q-learning to model bidding behaviour in electricity market simulation

While the choice between two pricing rules, namely Uniform pricing rule and Pay-as-bid pricing rule, has led to a continuous debate in the electricity market establishment process, little attention has been paid to the Vickrey pricing rule. This paper presents an agent-based model to examine the employment of Uniform and Vickrey pricing rules in a deregulated electricity market. Using Q-learning in repetitive trading process, generator agents learn the market characteristics and seek to maximise their revenue by exploring bidding strategies. A look up table is utilised to memorise agents' bidding experience that help the agents improve their strategies. Supply quantity withholding and generators' collusion phenomenon have been observed in this study under certain market arrangements. The implication of these two pricing rules on the total dispatch costs and generators' profit are discussed in this paper.

[1]  Yang Liu,et al.  A new Q-learning algorithm based on the metropolis criterion , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Ben J. A. Kröse,et al.  Learning from delayed rewards , 1995, Robotics Auton. Syst..

[3]  Leigh Tesfatsion,et al.  Market power and efficiency in a computational electricity market with discriminatory double-auction pricing , 2001, IEEE Trans. Evol. Comput..

[4]  E. Maskin.  Asymmetric Auctions , 2007 .

[5]  R. Baldick,et al.  Short-term electricity market auction game analysis: uniform and pay-as-bid pricing , 2004, IEEE Transactions on Power Systems.

[6]  Damien Ernst,et al.  Nash Equilibria and Reinforcement Learning for Active Decision Maker Modelling in Power Markets , 2004 .

[7]  Derek W. Bunn,et al.  Evaluating Individual Market Power in Electricity Markets via Agent-Based Simulation , 2003, Ann. Oper. Res..

[8]  Derek W. Bunn,et al.  Agent-based simulation-an application to the new electricity trading arrangements of England and Wales , 2001, IEEE Trans. Evol. Comput..

[9]  Richard D. Tabors,et al.  Uniform Pricing or Pay-as-Bid Pricing: A Dilemma for California and Beyond , 2001 .

[10]  S. Okuma,et al.  Multi-agent based experiments on uniform price and pay-as-bid electricity auction markets , 2004, 2004 IEEE International Conference on Electric Utility Deregulation, Restructuring and Power Technologies. Proceedings.

[11]  Simon Haykin,et al.  A dynamic channel assignment policy through Q-learning , 1999, IEEE Trans. Neural Networks.

[12]  Natalia Fabra,et al.  Designing Electricity Auctions , 2004 .

[13]  S. Mitchell,et al.  Comments Welcome , 1999 .

[14]  S. Rassenti,et al.  Discriminatory Price Auctions in Electricity Markets: Low Volatility at the Expense of High Price Levels , 2003 .

[15]  I. Watanabe,et al.  Agent-based simulation model of electricity market with stochastic unit commitment , 2004, 2004 International Conference on Probabilistic Methods Applied to Power Systems.