Optimal control in microgrid using multi-agent reinforcement learning.

This paper presents an improved reinforcement learning method to minimize electricity costs on the premise of satisfying the power balance and generation limit of units in a microgrid with grid-connected mode. Firstly, the microgrid control requirements are analyzed and the objective function of optimal control for microgrid is proposed. Then, a state variable "Average Electricity Price Trend" which is used to express the most possible transitions of the system is developed so as to reduce the complexity and randomicity of the microgrid, and a multi-agent architecture including agents, state variables, action variables and reward function is formulated. Furthermore, dynamic hierarchical reinforcement learning, based on change rate of key state variable, is established to carry out optimal policy exploration. The analysis shows that the proposed method is beneficial to handle the problem of "curse of dimensionality" and speed up learning in the unknown large-scale world. Finally, the simulation results under JADE (Java Agent Development Framework) demonstrate the validity of the presented method in optimal control for a microgrid with grid-connected mode.

[1]  M.P.F. Hommelberg,et al.  A novel architecture for real-time operation of multi-agent based coordination of demand and supply , 2008, 2008 IEEE Power and Energy Society General Meeting - Conversion and Delivery of Electrical Energy in the 21st Century.

[2]  Thillainathan Logenthiran,et al.  Multi-agent system for energy resource scheduling of integrated microgrids in a distributed system , 2011 .

[3]  Graham Coates,et al.  A Multi-Agent System for decentralised control of low voltage distribution networks , 2010, 45th International Universities Power Engineering Conference UPEC2010.

[4]  Jun Zeng,et al.  A multi-agent solution to energy management of distributed hybrid renewable energy generated system , 2009 .

[5]  Manfred Huber,et al.  Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies , 2003 .

[6]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[7]  D. Ernst,et al.  Power systems stability control: reinforcement learning framework , 2004, IEEE Transactions on Power Systems.

[8]  David A. Cartes,et al.  Distributed energy resource management in a smart grid by risk based auction strategy for profit maximization , 2010, IEEE PES General Meeting.

[9]  H. Morais,et al.  Distributed energy resources management with cyber-physical SCADA in the context of future smart grids , 2010, Melecon 2010 - 2010 15th IEEE Mediterranean Electrotechnical Conference.

[10]  Agostino Poggi,et al.  Developing Multi-agent Systems with JADE , 2007, ATAL.

[11]  Magdy M. A. Salama,et al.  Distributed generation technologies, definitions and benefits , 2004 .

[12]  N.N. Schulz,et al.  A Multi-Agent Solution to Distribution Systems Restoration , 2007, IEEE Transactions on Power Systems.

[13]  Jaap Gordijn,et al.  Business models for distributed generation in a liberalized market environment , 2007 .

[14]  S. Suryanarayanan,et al.  A framework for energy management in customer-driven microgrids , 2010, IEEE PES General Meeting.

[15]  Toshiyuki Ito,et al.  Application of Mobile Agent Technology to Power Generation Control in Microgrid Power System , 2009, 2009 Asia-Pacific Power and Energy Engineering Conference.

[16]  Ronald E. Parr,et al.  Hierarchical control and learning for markov decision processes , 1998 .

[17]  Wu Min.  An Improved Control Strategy of Load Distribution in an Autonomous Microgrid , 2011 .

[18]  T. Logenthiran,et al.  Multi-agent coordination for DER in MicroGrid , 2008, 2008 IEEE International Conference on Sustainable Energy Technologies.

[19]  S. Kennedy,et al.  A new wholesale bidding mechanism for enhanced demand response in smart grids , 2010, 2010 Innovative Smart Grid Technologies (ISGT).

[20]  S.D.J. McArthur,et al.  Multi-Agent Systems for Power Engineering Applications—Part I: Concepts, Approaches, and Technical Challenges , 2007, IEEE Transactions on Power Systems.

[21]  Thomas G. Dietterich.  Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition , 1999, J. Artif. Intell. Res..

[22]  S. Suryanarayanan,et al.  A conceptual framework of a hierarchically networked agent-based microgrid architecture , 2010, IEEE PES T&D 2010.

[23]  Thomas G. Dietterich.  An Overview of MAXQ Hierarchical Reinforcement Learning , 2000, SARA.

[24]  P. S. Nagendra Rao,et al.  A reinforcement learning approach to automatic generation control , 2002 .