(Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives

Abstract This paper reviews existing works on (deep) reinforcement learning considerations in electric power system control. The works are reviewed as they relate to electric power system operating states (normal, preventive, emergency, restorative) and control levels (local, household, microgrid, subsystem, wide-area). Due attention is paid to the control-related problems considerations (cyber-security, big data analysis, short-term load forecast, and composite load modelling). Observations from reviewed literature are drawn and perspectives discussed. In order to make the text compact and as easy as possible to read, the focus is only on the works published (or “in press”) in journals and books while conference publications are not included. Exceptions are several work available in open repositories likely to become journal publications in near future. Hopefully this paper could serve as a good source of information for all those interested in solving similar problems.

[1]  Ying Chen,et al.  Evaluation of Reinforcement Learning-Based False Data Injection Attack to Automatic Voltage Control , 2019, IEEE Transactions on Smart Grid.

[2]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[3]  Enrico Anderlini,et al.  Control of a Point Absorber Using Reinforcement Learning , 2016, IEEE Transactions on Sustainable Energy.

[4]  Tao Yu,et al.  Hierarchical correlated Q-learning for multi-layer optimal generation command dispatch , 2016 .

[5]  Yi Chai,et al.  An integrated critic-actor neural network for reinforcement learning with application of DERs control in grid frequency regulation , 2019, International Journal of Electrical Power & Energy Systems.

[6]  Peter Henderson,et al.  An Introduction to Deep Reinforcement Learning , 2018, Found. Trends Mach. Learn..

[7]  Xinping Guan,et al.  Risk-Averse Transmission Path Selection for Secure State Estimation in Power Systems , 2019, IEEE Internet of Things Journal.

[8]  Hanchen Xu,et al.  Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning , 2018, IEEE Transactions on Power Systems.

[9]  Tao Yu,et al.  Memetic reinforcement learning based maximum power point tracking design for PV systems under partial shading condition , 2019, Energy.

[10]  Anuradha M. Annaswamy,et al.  Controls for Smart Grids: Architectures and Applications , 2017, Proceedings of the IEEE.

[11]  Wenxin Liu,et al.  Q-Learning-Based Damping Control of Wide-Area Power Systems Under Cyber Uncertainties , 2018, IEEE Transactions on Smart Grid.

[12]  Mehdi Bagheri,et al.  Enhancing Power Quality in Microgrids With a New Online Control Strategy for DSTATCOM Using Reinforcement Learning Algorithm , 2018, IEEE Access.

[13]  Fankun Bu,et al.  Data-Driven Based Method for Power System Time-Varying Composite Load Modeling , 2019, 1905.02688.

[14]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[15]  Song Guo,et al.  A Survey on Energy Internet: Architecture, Approach, and Emerging Technologies , 2018, IEEE Systems Journal.

[16]  Philipp Hennig,et al.  Dual Control for Approximate Bayesian Reinforcement Learning , 2015, J. Mach. Learn. Res..

[17]  D. Ernst,et al.  Combining a stability and a performance-oriented control in power systems , 2005, IEEE Transactions on Power Systems.

[18]  Adeniyi A. Babalola,et al.  Reinforcement learning approach for congestion management and cascading failure prevention with experimental application , 2016 .

[19]  M. Glavic,et al.  Distributed Undervoltage Load Shedding , 2007, IEEE Transactions on Power Systems.

[20]  Minjie Zhang,et al.  A Hybrid Multiagent Framework With Q-Learning for Power Grid Systems Restoration , 2011, IEEE Transactions on Power Systems.

[21]  Zhen Yu,et al.  Online tuning of a supervisory fuzzy controller for low-energy building system using reinforcement learning , 2010 .

[22]  G. Burt,et al.  Comparing Policy Gradient and Value Function Based Reinforcement Learning Methods in Simulated Electrical Power Trade , 2012, IEEE Transactions on Power Systems.

[23]  Xiaoxin Zhou,et al.  Learning-coordinate fuzzy logic control of dynamic quadrature boosters in multi-machine power systems , 1999 .

[24]  Yan Xu,et al.  Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search , 2019, IEEE Transactions on Power Systems.

[25]  Rojan Bhattarai,et al.  Transient Stability Enhancement of Power Grid With Integrated Wide Area Control of Wind Farms and Synchronous Generators , 2017, IEEE Transactions on Power Systems.

[26]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[27]  Yue Tan,et al.  Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges , 2019, IEEE Communications Surveys & Tutorials.

[28]  Lucian Busoniu,et al.  Reinforcement learning for control: Performance, stability, and deep approximators , 2018, Annu. Rev. Control..

[29]  Javad Lavaei,et al.  Stability-Certified Reinforcement Learning: A Control-Theoretic Perspective , 2018, IEEE Access.

[30]  Chong Li,et al.  Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach , 2018, IEEE Transactions on Smart Grid.

[31]  C. Boutilier,et al.  Accelerating Reinforcement Learning through Implicit Imitation , 2003, J. Artif. Intell. Res..

[32]  Victor C. M. Leung,et al.  Performance Optimization for Blockchain-Enabled Industrial Internet of Things (IIoT) Systems: A Deep Reinforcement Learning Approach , 2019, IEEE Transactions on Industrial Informatics.

[33]  Kuang-Ching Wang,et al.  Review of Internet of Things (IoT) in Electric Power and Energy Systems , 2018, IEEE Internet of Things Journal.

[34]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[35]  Bart De Schutter,et al.  Reinforcement Learning and Dynamic Programming Using Function Approximators , 2010 .

[36]  Abhinav Gupta,et al.  Robust Adversarial Reinforcement Learning , 2017, ICML.

[37]  Johannes Fürnkranz,et al.  A Survey of Preference-Based Reinforcement Learning Methods , 2017, J. Mach. Learn. Res..

[38]  Ali Feliachi,et al.  Reinforcement learning based backstepping control of power system oscillations , 2009 .

[39]  Richard S. Sutton,et al.  Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning , 2020, AAAI.

[40]  Safety-Guided Deep Reinforcement Learning via Online Gaussian Process Estimation , 2019, ArXiv.

[41]  Yilu Liu,et al.  A Novel Equivalent Model of Active Distribution Networks Based on LSTM , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Warren B. Powell,et al.  Tutorial on Stochastic Optimization in Energy—Part I: Modeling and Policies , 2016, IEEE Transactions on Power Systems.

[43]  Frank L. Lewis,et al.  Optimal and Autonomous Control Using Reinforcement Learning: A Survey , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[44]  C. C. Liu,et al.  Adaptation in Load Shedding under Vulnerable Operation Conditions , 2002, IEEE Power Engineering Review.

[45]  Wail Gueaieb,et al.  Load frequency regulation for multi‐area power system using integral reinforcement learning , 2019, IET Generation, Transmission & Distribution.

[46]  Tao Yu,et al.  Artificial emotional reinforcement learning for automatic generation control of large-scale interconnected power grids , 2017 .

[47]  T. E. Dy Liacco,et al.  Real-time computer control of power systems , 1974 .

[48]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[49]  Ali Feliachi,et al.  Reinforcement learning tuned decentralized synergetic control of power systems , 2012 .

[50]  Damien Ernst,et al.  Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives , 2017 .

[51]  Tommaso Mannucci,et al.  Safe Exploration Algorithms for Reinforcement Learning Controllers , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[52]  Tao Yu,et al.  Lifelong Learning for Complementary Generation Control of Interconnected Power Grids With High-Penetration Renewables and EVs , 2018, IEEE Transactions on Power Systems.

[53]  Kamyar Azizzadenesheli,et al.  Efficient Exploration Through Bayesian Deep Q-Networks , 2018, 2018 Information Theory and Applications Workshop (ITA).

[54]  Xiangtian Zheng,et al.  Nested Reinforcement Learning Based Control for Protective Relays in Power Distribution Systems , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[55]  Mevludin Glavic,et al.  Design of a resistive brake controller for power system stability enhancement using reinforcement learning , 2005, IEEE Transactions on Control Systems Technology.

[56]  Wei Qiao,et al.  An Adaptive Network-Based Reinforcement Learning Method for MPPT Control of PMSG Wind Energy Conversion Systems , 2016, IEEE Transactions on Power Electronics.

[57]  Zhehan Yi,et al.  Reinforcement-Learning-Based Optimal Control of Hybrid Energy Storage Systems in Hybrid AC–DC Microgrids , 2019, IEEE Transactions on Industrial Informatics.

[58]  Wei Zhang,et al.  Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[59]  D. Ernst,et al.  Automatic learning of sequential decision strategies for dynamic security assessment and control , 2006, 2006 IEEE Power Engineering Society General Meeting.

[60]  Jianchun Peng,et al.  Multiobjective Reinforcement Learning-Based Intelligent Approach for Optimization of Activation Rules in Automatic Generation Control , 2019, IEEE Access.

[61]  P. S. Nagendra Rao,et al.  A reinforcement learning approach to automatic generation control , 2002 .

[62]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[63]  Simon Haykin,et al.  Cognitive Risk Control for Mitigating Cyber-Attack in Smart Grid , 2019, IEEE Access.

[64]  Huaguang Zhang,et al.  Fault-Tolerant Controller Design for a Class of Nonlinear MIMO Discrete-Time Systems via Online Reinforcement Learning Algorithm , 2016, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[65]  Javier García,et al.  A comprehensive survey on safe reinforcement learning , 2015, J. Mach. Learn. Res..

[66]  Douglas C. Hittle,et al.  Robust reinforcement learning control with static and dynamic stability , 2001 .

[67]  Chen-Ching Liu,et al.  The strategic power infrastructure defense (SPID) system. A conceptual design , 2000, IEEE Control Systems.

[68]  D. Ernst,et al.  Power systems stability control: reinforcement learning framework , 2004, IEEE Transactions on Power Systems.

[69]  N.D. Hatziargyriou,et al.  Reinforcement learning for reactive power control , 2004, IEEE Transactions on Power Systems.

[70]  Xuemin Zhang,et al.  Power System Security Under False Data Injection Attacks With Exploitation and Exploration Based on Reinforcement Learning , 2018, IEEE Access.

[71]  Louis Wehenkel,et al.  A reinforcement learning based discrete supplementary control for power system transient stability enhancement , 2005 .

[72]  Anuradha M. Annaswamy,et al.  Systems & Control for the future of humanity, research agenda: Current and future roles, impact and grand challenges , 2017, Annu. Rev. Control..

[73]  Adeniyi A. Babalola,et al.  Real-Time Cascading Failures Prevention Through MAS Algorithm and Immune System Reinforcement Learning , 2017 .

[74]  Haibo He,et al.  Q-Learning-Based Vulnerability Analysis of Smart Grid Against Sequential Topology Attacks , 2017, IEEE Transactions on Information Forensics and Security.

[75]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[76]  Qingyu Yang,et al.  Defending Against Data Integrity Attacks in Smart Grid: A Deep Reinforcement Learning-Based Approach , 2019, IEEE Access.

[77]  Tao Yu,et al.  R(λ) imitation learning for automatic generation control of interconnected power grids , 2012, Autom..

[78]  Jie Zhang,et al.  Reinforced Deterministic and Probabilistic Load Forecasting via $Q$ -Learning Dynamic Model Selection , 2020, IEEE Transactions on Smart Grid.

[79]  Alexander Apostolov,et al.  IEEE PSRC Report on Global Industry Experiences With System Integrity Protection Schemes (SIPS) , 2010, IEEE Transactions on Power Delivery.

[80]  Ali Feliachi,et al.  A Multiagent Design for Power Distribution Systems Automation , 2016, IEEE Transactions on Smart Grid.

[81]  Tao Yu,et al.  Stochastic Optimal Relaxed Automatic Generation Control in Non-Markov Environment Based on Multi-Step $Q(\lambda)$ Learning , 2011, IEEE Transactions on Power Systems.

[82]  D. Ernst,et al.  Approximate Value Iteration in the Reinforcement Learning Context. Application to Electrical Power System Control. , 2005 .

[83]  Munther A. Dahleh,et al.  Advancing systems and control research in the era of ML and AI , 2018, Annu. Rev. Control..

[84]  Wail Gueaieb,et al.  Model-Free Adaptive Learning Control Scheme for Wind Turbines with Doubly Fed Induction Generators , 2018 .

[85]  Stefan Schaal,et al.  A Generalized Path Integral Control Approach to Reinforcement Learning , 2010, J. Mach. Learn. Res..

[86]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[87]  D. Bertsekas.  Reinforcement Learning and Optimal ControlA Selective Overview , 2018 .

[88]  Jianhua Li,et al.  Big Data Analysis-Based Security Situational Awareness for Smart Grid , 2018, IEEE Transactions on Big Data.

[89]  D. Ernst,et al.  Transient stability emergency control combining open-loop and closed-loop techniques , 2003, 2003 IEEE Power Engineering Society General Meeting (IEEE Cat. No.03CH37491).

[90]  Christoph Adami.  Artificial intelligence: Robots with instincts , 2015, Nature.

[91]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[92]  Renke Huang,et al.  Adaptive Power System Emergency Control Using Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[93]  Sukumar Kamalasadan,et al.  Energy Function Inspired Value Priority Based Global Wide-Area Control of Power Grid , 2018, IEEE Transactions on Smart Grid.

[94]  Mohammad Moradzadeh,et al.  Application of reinforcement learning for generating optimal control signal to the IPFC for damping of low‐frequency oscillations , 2018 .

[95]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[96]  Osama Mohammed,et al.  Energy Storage Technologies for High-Power Applications , 2016, IEEE Transactions on Industry Applications.

[97]  Warren E. Dixon,et al.  Reinforcement Learning for Optimal Feedback Control , 2018 .

[98]  Zhigang Li,et al.  Equivalent modeling of active distribution network considering the spatial uncertainty of renewable energy resources , 2019, International Journal of Electrical Power & Energy Systems.

[99]  Robert Babuska,et al.  Experience Selection in Deep Reinforcement Learning for Control , 2018, J. Mach. Learn. Res..

[100]  Daniel Kuhn,et al.  RLOC: Neurobiologically Inspired Hierarchical Reinforcement Learning Algorithm for Continuous Control of Nonlinear Dynamical Systems , 2019, ArXiv.

[101]  K. W. Chan,et al.  Multi-Agent Correlated Equilibrium Q(λ) Learning for Coordinated Smart Generation Control of Interconnected Power Grids , 2015, IEEE Transactions on Power Systems.

[102]  Alireza Bakhshai,et al.  Intelligent Control of Grid-Connected Microgrids: An Adaptive Critic-Based Approach , 2015, IEEE Journal of Emerging and Selected Topics in Power Electronics.

[103]  Hassan Bevrani,et al.  Load–frequency control : a GA-based multi-agent reinforcement learning , 2010 .

[104]  Sukumar Kamalasadan,et al.  Design and Real-Time Implementation of Optimal Power System Wide-Area System-Centric Controller Based on Temporal Difference Learning , 2014, IEEE Transactions on Industry Applications.

[105]  Zhehan Yi,et al.  Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations , 2020, IEEE Transactions on Power Systems.

[106]  Xinping Guan,et al.  Antijamming Game Framework for Secure State Estimation in Power Systems , 2019, IEEE Transactions on Industrial Informatics.

[107]  M. Pirani,et al.  A systems and control perspective of CPS security , 2019, Annu. Rev. Control..

[108]  Mario Zanon,et al.  Data-Driven Economic NMPC Using Reinforcement Learning , 2019, IEEE Transactions on Automatic Control.

[109]  Bidyadhar Subudhi,et al.  An Adaptive Variable Leaky Least Mean Square Control Scheme for Grid Integration of a PV System , 2020, IEEE Transactions on Sustainable Energy.

[110]  Anil A. Bharath,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[111]  K. R. Padiyar,et al.  Power system dynamics : stability and control , 1996 .

[112]  P. S. Nagendra Rao,et al.  A neural network based automatic generation control design through reinforcement learning , 2006 .

[113]  Zhe Zhang,et al.  Reinforcement-Learning-Based Intelligent Maximum Power Point Tracking Control for Wind Energy Conversion Systems , 2015, IEEE Transactions on Industrial Electronics.

[114]  Geert Deconinck,et al.  Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning , 2017, CSEE Journal of Power and Energy Systems.

[115]  Chi K. Tse,et al.  Sequential topology recovery of complex power systems based on reinforcement learning , 2019 .

[116]  Ramtin Hadidi,et al.  Reinforcement Learning Based Real-Time Wide-Area Stabilizing Control Agents to Enhance Power System Stability , 2013, IEEE Transactions on Smart Grid.

[117]  Enrico Anderlini,et al.  Control of a Realistic Wave Energy Converter Model Using Least-Squares Policy Iteration , 2017, IEEE Transactions on Sustainable Energy.

[118]  Louis Wehenkel,et al.  Trajectory-Based Supplementary Damping Control for Power System Electromechanical Oscillations , 2014, IEEE Transactions on Power Systems.