Volt-VAR Control in Power Distribution Systems with Deep Reinforcement Learning

Volt-VAR control (VVC) plays an important role in enhancing energy efficiency, power quality, and reliability of electric power distribution systems by coordinating the operations of equipment such as voltage regulators, on-load tap changers, and capacitor banks. VVC not only keeps voltages in the distribution system within desirable ranges but also reduces system operation costs, which include network losses and equipment depreciation from wear and tear. In this paper, the deep reinforcement learning approach is taken to learn a VVC policy, which minimizes the total operation costs while satisfying the physical operation constraints. The VVC problem is formulated as a constrained Markov decision process and solved by two policy gradient methods, trust region policy optimization and constrained policy optimization. Numerical study results based on IEEE 4-bus and 13-bus distribution test feeders show that the policy gradient methods are capable of learning near-optimal solutions and determining control actions much faster than the optimization-based approaches.

[1]  P. McSharry,et al.  Short-Term Load Forecasting Methods: An Evaluation Based on European Data , 2007, IEEE Transactions on Power Systems.

[2]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[3]  C. Cañizares,et al.  Reactive Power and Voltage Control in Distribution Systems With Limited Switching Operations , 2009, IEEE Transactions on Power Systems.

[4]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[5]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[6]  Bikash C. Pal,et al.  A Two-Stage Chance Constrained Volt/Var Control Scheme for Active Distribution Networks With Nodal Power Uncertainties , 2019, IEEE Transactions on Power Systems.

[7]  Juan Li,et al.  Phase Identification in Electric Power Distribution Systems by Clustering of Smart Meter Data , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[8]  Hanchen Xu,et al.  Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning , 2018, IEEE Transactions on Power Systems.

[9]  Jianzhong Wu,et al.  Coordinated Control Method of Voltage and Reactive Power for Active Distribution Networks Based on Soft Open Point , 2017, IEEE Transactions on Sustainable Energy.

[10]  Chadi Assi,et al.  Volt-VAR Control Through Joint Optimization of Capacitor Bank Switching, Renewable Energy, and Home Appliances , 2018, IEEE Transactions on Smart Grid.

[11]  Hamed Ahmadi,et al.  A Framework for Volt-VAR Optimization in Distribution Systems , 2015, IEEE Transactions on Smart Grid.

[12]  W. H. Kersting,et al.  Radial distribution test feeders , 1991, 2001 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No.01CH37194).

[13]  Mehrdad Ehsani,et al.  Dynamic Reactive Power Control of Islanded Microgrids , 2013, IEEE Transactions on Power Systems.

[14]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[15]  W. Wang,et al.  Advanced Metering Infrastructure Data Driven Phase Identification in Smart Grid , 2017 .

[16]  Nanpeng Yu,et al.  A Comprehensive Evaluation of Supervised Machine Learning for the Phase Identification Problem , 2018 .

[17]  Boming Zhang,et al.  Robust Reactive Power Optimization and Voltage Control Method for Active Distribution Networks via Dual Time-scale Coordination , 2016, ArXiv.

[18]  Felix F. Wu,et al.  Network Reconfiguration in Distribution Systems for Loss Reduction and Load Balancing , 1989, IEEE Power Engineering Review.

[19]  David J. Hill,et al.  Multi-Timescale Coordinated Voltage/Var Control of High Renewable-Penetrated Distribution Systems , 2017, IEEE Transactions on Power Systems.

[20]  Wei Zhang,et al.  Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[21]  Jianhui Wang,et al.  MPC-Based Voltage/Var Optimization for Distribution Circuits With Distributed Generators and Exponential Load Models , 2014, IEEE Transactions on Smart Grid.

[22]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[23]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[24]  Pieter Abbeel,et al.  Constrained Policy Optimization , 2017, ICML.

[25]  N.D. Hatziargyriou,et al.  Reinforcement learning for reactive power control , 2004, IEEE Transactions on Power Systems.