A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning

The complexity of modern power grids keeps increasing due to the expansion of renewable energy resources and the requirement of fast demand responses, which results in a great challenge for conventional power grid control systems. Existing autonomous control approaches for the power grid requires an accurate system model and a powerful computational platform, which is difficult to scale up for the large-scale energy system with more control options and operating conditions. Facing these challenges, this article proposes a data-driven multi-agent power grid control scheme using a deep reinforcement learning (DRL) method. Specifically, the classic autonomous voltage control (AVC) problem is taken as an example and formulated as a Markov Game with a heuristic method to partition agents. Then, a multi-agent AVC (MA-AVC) algorithm based on a multi-agent deep deterministic policy gradient (MADDPG) method that features centralized training and decentralized execution is developed to solve the AVC problem. The proposed method can learn from scratch and gradually master the system operation rules by input and output data. In order to demonstrate the effectiveness of the proposed MA-AVC algorithm, comprehensive case studies are conducted on an Illinois 200-Bus system considering load/generation changes, N-1 contingencies, and weak centralized communication environment.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[3]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[4]  Bin Wang,et al.  Optimal Voltage Control of PJM Smart Transmission Grid: Study, Implementation, and Evaluation , 2013, IEEE Transactions on Smart Grid.

[5]  Wei Zhang,et al.  Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[6]  N.D. Hatziargyriou,et al.  Reinforcement learning for reactive power control , 2004, IEEE Transactions on Power Systems.

[7]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[8]  Zhiwei Wang,et al.  Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning , 2019, 2019 IEEE Power & Energy Society General Meeting (PESGM).

[9]  Zhehan Yi,et al.  Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations , 2020, IEEE Transactions on Power Systems.

[10]  Damien Ernst,et al.  Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives , 2017 .

[11]  Steven H. Low,et al.  Convex Relaxation of Optimal Power Flow—Part I: Formulations and Equivalence , 2014, IEEE Transactions on Control of Network Systems.

[12]  Hanchen Xu,et al.  Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning , 2018, IEEE Transactions on Power Systems.

[13]  Henrik Sandberg,et al.  A Survey of Distributed Optimization and Control Algorithms for Electric Power Systems , 2017, IEEE Transactions on Smart Grid.

[14]  Hongbin Sun,et al.  Review of Challenges and Research Opportunities for Voltage Control in Smart Grids , 2019, IEEE Transactions on Power Systems.

[15]  Nikos D. Hatziargyriou,et al.  Distributed and Decentralized Voltage Control of Smart Distribution Networks: Models, Methods, and Future Research , 2017, IEEE Transactions on Smart Grid.

[16]  Wei Shi,et al.  Distributed Voltage Control in Distribution Networks: Online and Robust Implementations , 2018, IEEE Transactions on Smart Grid.

[17]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[18]  Hao Jan Liu,et al.  Fast Local Voltage Control Under Limited Reactive Power: Optimality and Stability Analysis , 2015, IEEE Transactions on Power Systems.

[19]  Zhe Chen,et al.  Multi-Stage Optimization-Based Automatic Voltage Control Systems Considering Wind Power Forecasting Errors , 2017, IEEE Transactions on Power Systems.

[20]  Venkataramana Ajjarapu,et al.  Real-Time Local Volt/Var Control Under External Disturbances With High PV Penetration , 2017, IEEE Transactions on Smart Grid.

[21]  Na Li,et al.  Optimal Distributed Feedback Voltage Control Under Limited Reactive Power , 2018, IEEE Transactions on Power Systems.

[22]  Thomas J. Overbye,et al.  Grid Structural Characteristics as Validation Criteria for Synthetic Networks , 2017, IEEE Transactions on Power Systems.