PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems

We introduce PowerGym, an open-source reinforcement learning environment for Volt-Var control in power distribution systems. Following OpenAI Gym APIs, PowerGym targets minimizing power loss and voltage violations under physical networked constraints. PowerGym provides four distribution systems (13Bus, 34Bus, 123Bus, and 8500Node) based on IEEE benchmark systems and design variants for various control difficulties. To foster generalization, PowerGym offers a detailed customization guide for users working with their distribution systems. As a demonstration, we examine state-of-the-art reinforcement learning algorithms in PowerGym and validate the environment by studying controller behaviors.

[1]  Yang Liu,et al.  Continuous Multiagent Control using Collective Behavior Entropy for Large-Scale Home Energy Management , 2020, AAAI.

[2]  Ting-Han Fan,et al.  Soft Actor-Critic With Integer Actions , 2021, ArXiv.

[3]  Yan Xu,et al.  Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search , 2019, IEEE Transactions on Power Systems.

[4]  Anatolij Zubow,et al.  ns3-gym: Extending OpenAI Gym for Networking Research , 2018, ArXiv.

[5]  D. Ernst,et al.  Power systems stability control: reinforcement learning framework , 2004, IEEE Transactions on Power Systems.

[6]  Wei Wang,et al.  Volt-VAR Control in Power Distribution Systems with Deep Reinforcement Learning , 2019, 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm).

[7]  Lijun Chen,et al.  Equilibrium and dynamics of local voltage control in distribution systems , 2013, 52nd IEEE Conference on Decision and Control.

[8]  Yingchen Zhang,et al.  Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems , 2021, IEEE Transactions on Smart Grid.

[9]  Isabelle Guyon,et al.  Learning to run a Power Network Challenge: a Retrospective Analysis , 2020, NeurIPS.

[10]  S. Kakade,et al.  Reinforcement Learning: Theory and Algorithms , 2019 .

[11]  Deunsol Yoon,et al.  Winning the L2RPN Challenge: Power Grid Management via Semi-Markov Afterstate Actor-Critic , 2021, ICLR.

[12]  Nanpeng Yu,et al.  Deep Reinforcement Learning in Power Distribution Systems: Overview, Challenges, and Opportunities , 2021, 2021 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT).

[13]  Robert C. Qiu,et al.  Deep reinforcement learning for power system: An overview , 2019, CSEE Journal of Power and Energy Systems.

[14]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[15]  R. C. Dugan,et al.  The IEEE 8500-node test feeder , 2010, IEEE PES T&D 2010.

[16]  Henry Zhu,et al.  Soft Actor-Critic Algorithms and Applications , 2018, ArXiv.

[17]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[18]  Wei Wang,et al.  Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems , 2020, IEEE Transactions on Smart Grid.

[19]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[20]  Yasuhiro Fujita,et al.  ChainerRL: A Deep Reinforcement Learning Library , 2019, J. Mach. Learn. Res..

[21]  Ufuk Topcu,et al.  Exact Convex Relaxation of Optimal Power Flow in Radial Networks , 2013, IEEE Transactions on Automatic Control.

[22]  Isabelle Guyon,et al.  Learning to run a power network challenge for training topology controllers , 2019, Electric Power Systems Research.

[23]  Yuval Tassa,et al.  MuJoCo: A physics engine for model-based control , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  Dario Amodei,et al.  Benchmarking Safe Exploration in Deep Reinforcement Learning , 2019 .

[25]  Dewen Hu,et al.  Multiobjective Reinforcement Learning: A Comprehensive Overview , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[26]  A. Bose,et al.  Optimal power flow based on successive linear approximation of power flow equations , 2016 .

[27]  Mengdi Wang,et al.  Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation , 2020, ICML.