A reinforcement learning algorithm developed to model GenCo strategic bidding behavior in multidimensional and continuous state and action spaces

The electricity market has provided a complex economic environment, and consequently has increased the requirement for advancement of learning methods. In the agent-based modeling and simulation framework of this economic system, the generation company's decision-making is modeled using reinforcement learning. Existing learning methods that model the generation company's strategic bidding behavior are not adapted to the non-stationary and non-Markovian environment involving multidimensional and continuous state and action spaces. This paper proposes a reinforcement learning method to overcome these limitations. The proposed method discovers the input space structure through the self-organizing map, exploits learned experience through Roth-Erev reinforcement learning and explores through the actor critic map. Simulation results from experiments show that the proposed method outperforms Simulated Annealing Q-Learning and Variant Roth-Erev reinforcement learning. The proposed method is a step towards more realistic agent learning in Agent-based Computational Economics.

[1]  Fushuan Wen,et al.  Impacts of emission trading and renewable energy support schemes on electricity market operation , 2011 .

[2]  Yang Liu,et al.  A new Q-learning algorithm based on the metropolis criterion , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  A. Roth,et al.  Learning in Extensive-Form Games: Experimental Data and Simple Dynamic Models in the Intermediate Term* , 1995 .

[4]  Isabel Praça,et al.  MASCEM: Electricity Markets Simulation with Strategic Agents , 2011, IEEE Intelligent Systems.

[5]  L. Tesfatsion,et al.  Dynamic Testing of Wholesale Power Market Designs: An Open-Source Agent-Based Framework , 2007 .

[6]  Reza Safabakhsh,et al.  Continuous state/action reinforcement learning: A growing self-organizing map approach , 2011, Neurocomputing.

[7]  L. Tesfatsion,et al.  Separation and volatility of locational marginal prices in restructured wholesale power markets , 2009 .

[8]  Deddy P. Koesrindartoto.  Discrete Double Auctions with Artificial Adaptive Agents: A Case Study of an Electricity Market Using a Double Auction Simulator , 2002 .

[9]  Hado van Hasselt,et al.  Reinforcement Learning in Continuous State and Action Spaces , 2012, Reinforcement Learning.

[10]  Mohammad Shahidehpour,et al.  SecurityConstrained Unit Commitment , 2002 .

[11]  Vladimir Koritarov,et al.  An agent-based approach to modeling interactions between emission market and electricity market , 2009, 2009 IEEE Power & Energy Society General Meeting.

[12]  Guoqing Zhang,et al.  The electronic capacitive voltage transformers error characteristics research and parameter optimization design , 2009, 2009 IEEE Power & Energy Society General Meeting.

[13]  P. Tankov,et al.  MULTI-FACTOR JUMP-DIFFUSION MODELS OF ELECTRICITY PRICES , 2008 .

[14]  A. Roth,et al.  Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria , 1998 .

[15]  Charles J. Gieseler,et al.  A Java Reinforcement Learning Module for the Recursive Porous Agent Simulation Toolkit: facilitating study and experimentation with reinforcement learning in social science multi-agent simulations. , 2005 .

[16]  Ashkan Rahimi-Kian,et al.  NetPMS: an internet-based power market simulator for educational purposes , 2012 .

[17]  A.C. Tellidou,et al.  Agent-Based Analysis of Capacity Withholding and Tacit Collusion in Electricity Markets , 2007, IEEE Transactions on Power Systems.

[18]  L. Tesfatsion,et al.  An Agent-Based Test Bed Study of Wholesale Power Market Performance Measures , 2008, IEEE Computational Intelligence Magazine.

[19]  Habib Rajabi Mashhadi,et al.  An Adaptive $Q$-Learning Algorithm Developed for Agent-Based Computational Modeling of Electricity Market , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[20]  Derek W. Bunn,et al.  Agent-based simulation-an application to the new electricity trading arrangements of England and Wales , 2001, IEEE Trans. Evol. Comput..

[21]  Sishaj P. Simon,et al.  Nodal ant colony optimization for solving profit based unit commitment problem for GENCOs , 2012, Appl. Soft Comput..

[22]  Leigh Tesfatsion,et al.  Market power and efficiency in a computational electricity market with discriminatory double-auction pricing , 2001, IEEE Trans. Evol. Comput..

[23]  M. Sheikh-El-Eslami,et al.  Emergence of capacity withholding: an agent-based simulation of a double price cap electricity market , 2012 .

[24]  Leigh Tesfatsion,et al.  DC Optimal Power Flow Formulation and Solution Using QuadProgJ , 2006 .

[25]  J. Price,et al.  Evaluation of Market Rules Using a Multi-Agent System Method , 2010, IEEE Transactions on Power Systems.

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Jarrod Trevathan,et al.  A SOFTWARE ARCHITECTURE FOR CONTINUOUS DOUBLE AUCTIONS , 2007 .

[28]  Ly Fie Sugianto,et al.  Using Q-learning to model bidding behaviour in electricity market simulation , 2011, 2011 IEEE Symposium on Computational Intelligence in Multicriteria Decision-Making (MDCM).