A reinforcement learning framework for optimal operation and maintenance of power grids

Abstract We develop a Reinforcement Learning framework for the optimal management of the operation and maintenance of power grids equipped with prognostics and health management capabilities. Reinforcement learning exploits the information about the health state of the grid components. Optimal actions are identified maximizing the expected profit, considering the aleatory uncertainties in the environment. To extend the applicability of the proposed approach to realistic problems with large and continuous state spaces, we use Artificial Neural Networks (ANN) tools to replace the tabular representation of the state-action value function. The non-tabular Reinforcement Learning algorithm adopting an ANN ensemble is designed and tested on the scaled-down power grid case study, which includes renewable energy sources, controllable generators, maintenance delays and prognostics and health management devices. The method strengths and weaknesses are identified by comparison to the reference Bellman’s optimally. Results show good approximation capability of Q-learning with ANN, and that the proposed framework outperforms expert-based solutions to grid operation and maintenance management.

[1]  Csaba Szepesvári,et al.  Algorithms for Reinforcement Learning , 2010, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[2]  Enrico Zio,et al.  Optimal allocation of prognostics and health management capabilities to improve the reliability of a power transmission network , 2019, Reliab. Eng. Syst. Saf..

[3]  P Jarman,et al.  Creepage discharge on insulation barriers in aged power transformers , 2010, IEEE Transactions on Dielectrics and Electrical Insulation.

[4]  Seung Ho Hong,et al.  A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach , 2018, Applied Energy.

[5]  Edoardo Patelli,et al.  Assessment of power grid vulnerabilities accounting for stochastic loads and model imprecision , 2018, International Journal of Electrical Power & Energy Systems.

[6]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[7]  E. A. Jasmin,et al.  Reinforcement Learning approaches to Economic Dispatch problem , 2011 .

[8]  R. Rocchetta,et al.  A reinforcement learning framework for optimisation of power grid operations and maintenance , 2018 .

[9]  Martin A. Riedmiller.  Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method , 2005, ECML.

[10]  T. Das,et al.  A Reinforcement Learning Model to Assess Market Power Under Auction-Based Energy Pricing , 2007, IEEE Transactions on Power Systems.

[11]  Seung Ho Hong,et al.  Incentive-based demand response for smart grid with reinforcement learning and deep neural network , 2019, Applied Energy.

[12]  Damien Ernst,et al.  Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives , 2017 .

[13]  Shie Mannor,et al.  Reinforcement learning in the presence of rare events , 2008, ICML '08.

[14]  Rojan Bhattarai,et al.  Transient Stability Enhancement of Power Grid With Integrated Wide Area Control of Wind Farms and Synchronous Generators , 2017, IEEE Transactions on Power Systems.

[15]  Enrico Zio,et al.  Reinforcement learning for microgrid energy management , 2013 .

[16]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[17]  Tianshu Wei,et al.  Deep reinforcement learning: Framework, applications, and embedded implementations: Invited paper , 2017, 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[18]  Ashraf A. El Damatty,et al.  Review on dynamic and quasi-static buffeting response of transmission lines under synoptic and non-synoptic winds , 2016 .

[19]  Jasmin E.A.,et al.  Reinforcement Learning Solution for Unit Commitment Problem through Pursuit Method , 2009, 2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies.

[20]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[21]  Jie Song,et al.  Deep Reinforcement Leaming for Short-term Voltage Control by Dynamic Load Shedding in China Southem Power Grid , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[22]  Eitan Gross.  On the Bellman’s principle of optimality , 2016 .

[23]  Seiichi Shin,et al.  Smart Grid Optimization by Deep Reinforcement Learning over Discrete and Continuous Action Space , 2018, 2018 IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC) (A Joint Conference of 45th IEEE PVSC, 28th PVSEC & 34th EU PVSEC).

[24]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[25]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[26]  Louis Wehenkel,et al.  A reinforcement learning based discrete supplementary control for power system transient stability enhancement , 2005 .

[27]  Rahul Goyal,et al.  Review of hydrodynamics instabilities in Francis turbine during off-design and transient operations , 2018 .

[28]  Enrico Zio,et al.  A Markov decision process framework for optimal operation of monitored multi-state systems , 2018 .

[29]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[30]  Marco Levorato,et al.  Residential Demand Response Using Reinforcement Learning , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[31]  Enrico Zio,et al.  Homogeneous Continuous-Time, Finite-State Hidden Semi-Markov Modeling for Enhancing Empirical Classification System Diagnostics of Industrial Components , 2018 .

[32]  P. S. Nagendra Rao,et al.  A reinforcement learning approach to automatic generation control , 2002 .

[33]  Magdi S. Mahmoud,et al.  Adaptive intelligent techniques for microgrid control systems: A survey , 2017 .

[34]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Xiaoliang Ma,et al.  Hierarchical multi-agent control of traffic lights based on collective learning , 2018, Eng. Appl. Artif. Intell..

[36]  Wang,et al.  Review of road traffic control strategies , 2003, Proceedings of the IEEE.

[37]  Etienne Perot,et al.  Deep Reinforcement Learning framework for Autonomous Driving , 2017, Autonomous Vehicles and Machines.

[38]  T. P. Imthias Ahamed,et al.  Reinforcement Learning Solution for Unit Commitment Problem through Pursuit Method , 2009 .

[39]  N.D. Hatziargyriou,et al.  Reinforcement learning for reactive power control , 2004, IEEE Transactions on Power Systems.