Stability Constrained Reinforcement Learning for Real-Time Voltage Control

Deep reinforcement learning (RL) has been recognized as a promising tool to address the challenges in real-time control of power systems. However, its deployment in real-world power systems has been hindered by a lack of formal stability and safety guarantees. In this paper, we propose a stability constrained reinforcement learning method for real-time voltage control in distribution grids and we prove that the proposed approach provides a formal voltage stability guarantee. The key idea underlying our approach is an explicitly constructed Lyapunov function that certifies stability. We demonstrate the effectiveness of the approach in case studies, where the proposed method can reduce the transient control cost by more than 30\% and shorten the response time by a third compared to a widely used linear policy, while always achieving voltage stability. In contrast, standard RL methods often fail to achieve voltage stability.

[1]  Andreas Krause,et al.  The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems , 2018, CoRL.

[2]  Haotian Liu,et al.  Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks , 2020, IEEE Transactions on Smart Grid.

[3]  J. Zico Kolter,et al.  Learning Stable Deep Dynamics Models , 2020, NeurIPS.

[4]  Ofir Nachum,et al.  A Lyapunov-based Approach to Safe Reinforcement Learning , 2018, NeurIPS.

[5]  Felix F. Wu,et al.  Network Reconfiguration in Distribution Systems for Loss Reduction and Load Balancing , 1989, IEEE Power Engineering Review.

[6]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[7]  Andrew G. Barto,et al.  Lyapunov Design for Safe Reinforcement Learning , 2003, J. Mach. Learn. Res..

[8]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[9]  Di Shi,et al.  A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning , 2020, IEEE Transactions on Power Systems.

[10]  Na Li,et al.  Optimal Distributed Feedback Voltage Control Under Limited Reactive Power , 2018, IEEE Transactions on Power Systems.

[11]  Munther A. Dahleh,et al.  Real-time decentralized voltage control in distribution networks , 2014, 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[12]  Scalable Voltage Control using Structure-Driven Hierarchical Deep Reinforcement Learning , 2021, ArXiv.

[13]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[14]  Wei Wang,et al.  Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems , 2020, IEEE Transactions on Smart Grid.

[15]  Wei Shi,et al.  Hybrid Voltage Control in Distribution Networks Under Limited Communication Rates , 2017, IEEE Transactions on Smart Grid.

[16]  Wei Wang,et al.  Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks , 2020, IEEE Transactions on Smart Grid.

[17]  Lin Gao,et al.  Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks , 2020 .

[18]  Tao Liu,et al.  Fast Distributed Reactive Power Control for Voltage Regulation in Distribution Networks , 2019, IEEE Transactions on Power Systems.

[19]  Baosen Zhang,et al.  Data-Driven Optimal Voltage Regulation Using Input Convex Neural Networks , 2020 .

[20]  Georgios B. Giannakis,et al.  Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[21]  Yingchen Zhang,et al.  Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems , 2021, IEEE Transactions on Smart Grid.

[22]  Baosen Zhang,et al.  Reinforcement Learning for Optimal Frequency Control: A Lyapunov Approach , 2020, ArXiv.

[23]  Zhehan Yi,et al.  Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations , 2020, IEEE Transactions on Power Systems.

[24]  Baosen Zhang,et al.  A local control approach to voltage regulation in distribution networks , 2013, 2013 North American Power Symposium (NAPS).

[25]  Ruggero Carli,et al.  The value of communication in the voltage regulation problem , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[26]  Joseph Sill,et al.  Monotonic Networks , 1997, NIPS.

[27]  Pierluigi Siano,et al.  A Survey on Power System Blackout and Cascading Events: Research Motivations and Challenges , 2019, Energies.

[28]  Hanchen Xu,et al.  Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning , 2018, IEEE Transactions on Power Systems.

[29]  Shaoshuai Mou,et al.  Neural Certificates for Safe Control Policies , 2020, ArXiv.

[30]  Marina Velikova,et al.  Monotone and Partially Monotone Neural Networks , 2010, IEEE Transactions on Neural Networks.

[31]  Sicun Gao,et al.  Neural Lyapunov Control , 2020, NeurIPS.

[32]  Sergey Levine,et al.  QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation , 2018, CoRL.

[33]  Weiping Li,et al.  Applied Nonlinear Control , 1991 .

[34]  Xianzhuo Sun,et al.  Two-Stage Volt/Var Control in Active Distribution Networks With Multi-Agent Deep Reinforcement Learning Method , 2021, IEEE Transactions on Smart Grid.

[35]  Qi Huang,et al.  A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters , 2020, IEEE Transactions on Power Systems.

[36]  Na Li,et al.  Distributed Optimal Voltage Control With Asynchronous and Delayed Communication , 2019, IEEE Transactions on Smart Grid.

[37]  Steven H. Low,et al.  Optimal inverter VAR control in distribution systems with high PV penetration , 2011, 2012 IEEE Power and Energy Society General Meeting.

[38]  Ruggero Carli,et al.  Distributed Reactive Power Feedback Control for Voltage Regulation and Loss Minimization , 2013, IEEE Transactions on Automatic Control.

[39]  Yang Weng,et al.  Distributed Energy Resources Topology Identification via Graphical Modeling , 2017, IEEE Transactions on Power Systems.

[40]  Guannan Qu,et al.  Reinforcement Learning for Decision-Making and Control in Power Systems: Tutorial, Review, and Vision , 2021, ArXiv.

[41]  Hao Jan Liu,et al.  Fast Local Voltage Control Under Limited Reactive Power: Optimality and Stability Analysis , 2015, IEEE Transactions on Power Systems.