Deep Reinforcement Learning for Power System Applications: An Overview

Due to increasing complexity, uncertainty and data dimensions in power systems, conventional methods often meet bottlenecks when attempting to solve decision and control problems. Therefore, data-driven methods toward solving such problems are being extensively studied. Deep reinforcement learning (DRL) is one of these data-driven methods and is regarded as real artificial intelligence (AI). DRL is a combination of deep learning (DL) and reinforcement learning (RL). This field of research has been applied to solve a wide range of complex sequential decision-making problems, including those in power systems. This paper firstly reviews the basic ideas, models, algorithms and techniques of DRL. Applications in power systems such as energy management, demand response, electricity market, operational control, and others are then considered. In addition, recent advances in DRL including the combination of RL with other classical methods, and the prospect and challenges of applications in power systems are also discussed.

[1]  Yan Xu,et al.  Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search , 2019, IEEE Transactions on Power Systems.

[2]  K. R. Sudha,et al.  Dynamic Stability Margin Evaluation of Multi-machine Power Systems Using Genetic Algorithm , 2018 .

[3]  George A. Vouros,et al.  Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids , 2018, Applied Energy.

[4]  Pierre Pinson,et al.  An Integrated Market for Electricity and Natural Gas Systems with Stochastic Power Producers , 2018, Eur. J. Oper. Res..

[5]  Wencong Su,et al.  Indirect Customer-to-Customer Energy Trading With Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[6]  Zhen Ni,et al.  A Multistage Game in Smart Grid Security: A Reinforcement Learning Solution , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[7]  E. Firmansyah,et al.  Analyze and optimization of genetic algorithm implemented on maximum power point tracking technique for PV system , 2017, 2017 International Seminar on Application for Technology of Information and Communication (iSemantic).

[8]  Tao Chen,et al.  Local Energy Trading Behavior Modeling With Deep Reinforcement Learning , 2018, IEEE Access.

[9]  Hongwen He,et al.  Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus , 2019, Applied Energy.

[10]  Long Bao Le,et al.  A Bidding Strategy for Virtual Power Plants With the Intraday Demand Response Exchange Market Using the Stochastic Programming , 2018, IEEE Transactions on Industry Applications.

[11]  Zhen Ni,et al.  A Study of Linear Programming and Reinforcement Learning for One-Shot Game in Smart Grid Security , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[12]  Muhammad Babar,et al.  The Evaluation of Agile Demand Response: An Applied Methodology , 2018, IEEE Transactions on Smart Grid.

[13]  Juan M. Corchado,et al.  Decentralized Control of DR Using a Multi-agent Method , 2018 .

[14]  Maria Lorena Tuballa,et al.  A review of the development of Smart Grid technologies , 2016 .

[15]  Seiichi Shin,et al.  Smart Grid Optimization by Deep Reinforcement Learning over Discrete and Continuous Action Space , 2018, 2018 IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC) (A Joint Conference of 45th IEEE PVSC, 28th PVSEC & 34th EU PVSEC).

[16]  Aitor Saenz-Aguirre,et al.  Artificial Neural Network Based Reinforcement Learning for Wind Turbine Yaw Control , 2019, Energies.

[17]  Geert Deconinck,et al.  Battery Scheduling in a Residential Multi-Carrier Energy System Using Reinforcement Learning , 2018, 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm).

[18]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[19]  Tao Yu,et al.  Bacteria Foraging Reinforcement Learning for Risk-Based Economic Dispatch via Knowledge Transfer , 2017 .

[20]  C. Lindsay Anderson,et al.  Stochastic dynamic programming approach to managing power system uncertainty with distributed storage , 2018, Comput. Manag. Sci..

[21]  Jie Zhang,et al.  Inference Aided Reinforcement Learning for Incentive Mechanism Design in Crowdsourcing , 2018, NeurIPS.

[22]  Xiaoqing Han,et al.  Review on the research and practice of deep learning and reinforcement learning in smart grids , 2018, CSEE Journal of Power and Energy Systems.

[23]  Tao Yu,et al.  Deep transfer Q-learning with virtual leader-follower for supply-demand Stackelberg game of smart grid , 2017 .

[24]  Shabbir Ahmed,et al.  The Value of Multi-Stage Stochastic Programming in Risk-Averse Unit Commitment Under Uncertainty , 2019, IEEE Transactions on Power Systems.

[25]  Xingxing Zhang,et al.  A review of reinforcement learning methodologies for controlling occupant comfort in buildings , 2019, Sustainable Cities and Society.

[26]  Mihaela van der Schaar,et al.  Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning , 2016, IEEE Transactions on Smart Grid.

[27]  Mohamad Assaad,et al.  Multi -Agent Deep Reinforcement Learning based Power Control for Large Energy Harvesting Networks , 2019, 2019 International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOPT).

[28]  Peter Henderson,et al.  An Introduction to Deep Reinforcement Learning , 2018, Found. Trends Mach. Learn..

[29]  Tingwen Huang,et al.  A survey on distributed optimisation approaches and applications in smart grids , 2018, Journal of Control and Decision.

[30]  Razvan Pascanu,et al.  Sim-to-Real Robot Learning from Pixels with Progressive Nets , 2016, CoRL.

[31]  Pierluigi Siano,et al.  Demand response and smart grids—A survey , 2014 .

[32]  Damien Ernst,et al.  Deep Reinforcement Learning Solutions for Energy Microgrids Management , 2016 .

[33]  Guoyuan Wu,et al.  Deep reinforcement learning enabled self-learning control for energy efficient driving , 2019, Transportation Research Part C: Emerging Technologies.

[34]  Peter Stone,et al.  Reinforcement learning , 2019, Scholarpedia.

[35]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[36]  Jun Hao.  Multi-agent Reinforcement Learning Embedded Game for the Optimization of Building Energy Control and Power System Planning , 2019, ArXiv.

[37]  Vincent François-Lavet,et al.  Contributions to deep reinforcement learning and its applications in smartgrids , 2017 .

[38]  Siliang Lu,et al.  A DEEP REINFORCEMENT LEARNING APPROACH TO USINGWHOLE BUILDING ENERGYMODEL FOR HVAC OPTIMAL CONTROL , 2018 .

[39]  Damien Ernst,et al.  Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives , 2017 .

[40]  Robert C. Qiu,et al.  Smart Grid and Big Data: Theory and Practice , 2015 .

[41]  Lei Xi,et al.  Smart generation control based on deep reinforcement learning with the ability of action self-optimization , 2018 .

[42]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[43]  Zhiwei Wang,et al.  Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning , 2019, 2019 IEEE Power & Energy Society General Meeting (PESGM).

[44]  Ming Ding,et al.  A Review on Multi-Agent Technology in Micro-Grid Control , 2018 .

[45]  Yadong Liu,et al.  Massive Streaming PMU Data Modelling and Analytics in Smart Grid State Evaluation based on Multiple High-Dimensional Covariance Test , 2016, IEEE Transactions on Big Data.

[46]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[47]  Mohamed Benbouzid,et al.  Microgrids energy management systems: A critical review on methods, solutions, and prospects , 2018, Applied Energy.

[48]  Choong Seon Hong,et al.  When Edge Computing Meets Microgrid: A Deep Reinforcement Learning Approach , 2019, IEEE Internet of Things Journal.

[49]  Xiaofeng Liao,et al.  Reinforcement Learning for Constrained Energy Trading Games With Incomplete Information , 2017, IEEE Transactions on Cybernetics.

[50]  Geert Deconinck,et al.  Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning , 2017, CSEE Journal of Power and Energy Systems.

[51]  Geert Deconinck,et al.  Self-learning agent for battery energy management in a residential microgrid , 2018, 2018 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe).

[52]  Dale Schuurmans,et al.  Bridging the Gap Between Value and Policy Based Reinforcement Learning , 2017, NIPS.

[53]  Guoyuan Wu,et al.  Deep reinforcement learning-based vehicle energy efficiency autonomous learning system , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[54]  Richard Socher,et al.  The Natural Language Decathlon: Multitask Learning as Question Answering , 2018, ArXiv.

[55]  Damien Ernst,et al.  Real-Time Bidding Strategies from Micro-Grids Using Reinforcement Learning , 2018 .

[56]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[57]  Seung Ho Hong,et al.  A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach , 2018, Applied Energy.

[58]  Xinghuo Yu,et al.  An Overall Distribution Particle Swarm Optimization MPPT Algorithm for Photovoltaic System Under Partial Shading , 2019, IEEE Transactions on Industrial Electronics.

[59]  Liang Xiao,et al.  Reinforcement Learning-based Energy Trading for Microgrids , 2018, ArXiv.

[60]  Qian Ai,et al.  Big data analysis of power grid from random matrix theory , 2016 .

[61]  Jianfeng Chen,et al.  Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel , 2017 .

[62]  Na Li,et al.  Real-Time Energy Management in Microgrids , 2017, IEEE Transactions on Smart Grid.

[63]  Chenming Li,et al.  Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning , 2018 .

[64]  Qian Ai,et al.  A Big Data Architecture Design for Smart Grids Based on Random Matrix Theory , 2015, IEEE Transactions on Smart Grid.

[65]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[66]  Hanchen Xu,et al.  Deep Reinforcement Learning for Joint Bidding and Pricing of Load Serving Entity , 2019, IEEE Transactions on Smart Grid.

[67]  Tao Yu,et al.  A new generation of AI: A review and perspective on machine learning technologies applied to smart energy and electric power systems , 2019, International Journal of Energy Research.

[68]  Yu Tao.  Application and development of reinforcement learning theory in power systems , 2009 .

[69]  Pierre Baldi,et al.  From Reinforcement Learning to Deep Reinforcement Learning: An Overview , 2017, Braverman Readings in Machine Learning.

[70]  William Yang Wang,et al.  Deep Reinforcement Learning for NLP , 2018, ACL.

[71]  Ying Chen,et al.  Evaluation of Reinforcement Learning-Based False Data Injection Attack to Automatic Voltage Control , 2019, IEEE Transactions on Smart Grid.

[72]  Seung Ho Hong,et al.  Incentive-based demand response for smart grid with reinforcement learning and deep neural network , 2019, Applied Energy.

[73]  Dan-Lu Wang,et al.  Optimal Energy Routing Design in Energy Internet with Multiple Energy Routing Centers Using Artificial Neural Network-Based Reinforcement Learning Method , 2019 .

[74]  Robert C. Qiu,et al.  Spatiotemporal Big Data Analysis for Smart Grids Based on Random Matrix Theory , 2018 .

[75]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[76]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[77]  Xiangyu Zhang,et al.  A Data-driven Approach for Coordinating Air Conditioning Units in Buildings during Demand Response Events , 2019 .

[78]  Tapan Kumar Saha,et al.  Minimum Synchronous Inertia Requirement of Renewable Power Systems , 2018, IEEE Transactions on Power Systems.

[79]  Leslie K. Norford,et al.  Optimal control of HVAC and window systems for natural ventilation through reinforcement learning , 2018, Energy and Buildings.

[80]  Hanchen Xu,et al.  Arbitrage of Energy Storage in Electricity Markets with Deep Reinforcement Learning , 2019, ArXiv.

[81]  Vinatha U,et al.  Optimized design of collector topology for offshore wind farm based on ant colony optimization with multiple travelling salesman problem , 2018 .

[82]  Hao Liang,et al.  Distributed Economic Dispatch in Microgrids Based on Cooperative Reinforcement Learning , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[83]  Bart De Schutter,et al.  Reinforcement Learning and Dynamic Programming Using Function Approximators , 2010 .

[84]  Adriana Chis,et al.  Reinforcement Learning-Based Plug-in Electric Vehicle Charging With Forecasted Price , 2017, IEEE Transactions on Vehicular Technology.

[85]  Tao Yu,et al.  Deep Forest Reinforcement Learning for Preventive Strategy Considering Automatic Generation Control in Large-Scale Interconnected Power Systems , 2018 .

[86]  Haibo He,et al.  Residential Energy Management with Deep Reinforcement Learning , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[87]  Liang Xiao,et al.  Energy Trading Game for Microgrids Using Reinforcement Learning , 2017, GAMENETS.

[88]  Jakub W. Pachocki,et al.  Learning dexterous in-hand manipulation , 2018, Int. J. Robotics Res..

[89]  Tao Yu,et al.  Design of a Novel Smart Generation Controller Based on Deep Q Learning for Large-Scale Interconnected Power System , 2018 .

[90]  Lei Yang,et al.  Reinforcement Learning Based Pricing for Demand Response , 2018, 2018 IEEE International Conference on Communications Workshops (ICC Workshops).

[91]  Symeon Papavassiliou,et al.  Demand Response Management in Smart Grid Networks: a Two-Stage Game-Theoretic Learning-Based Approach , 2018, Mob. Networks Appl..

[92]  Prodromos Daoutidis,et al.  Microgrid/Macrogrid Energy Exchange: A Novel Market Structure and Stochastic Scheduling , 2017, IEEE Transactions on Smart Grid.

[93]  Tao Yu,et al.  Consensus Transfer ${Q}$ -Learning for Decentralized Generation Command Dispatch Based on Virtual Generation Tribe , 2018, IEEE Transactions on Smart Grid.

[94]  E. A. Jasmin,et al.  Residential Load Scheduling With Renewable Generation in the Smart Grid: A Reinforcement Learning Approach , 2019, IEEE Systems Journal.

[95]  H. Vincent Poor,et al.  Smarter energy : from smart metering to the smart grid , 2016 .

[96]  Geert Deconinck,et al.  Battery Energy Management in a Microgrid Using Batch Reinforcement Learning , 2017 .

[97]  B. Anand,et al.  Ant Colony Optimization Based Load Frequency Control of Multi-area Interconnected Thermal Power System with Governor Dead-Band Nonlinearity , 2018 .

[98]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[99]  Jean-Laurent Duchaud,et al.  Power ramp-rate control algorithm with optimal State of Charge reference via Dynamic Programming , 2018 .

[100]  Csaba Szepesvári,et al.  Algorithms for Reinforcement Learning , 2010, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[101]  Tingwen Huang,et al.  Reinforcement Learning in Energy Trading Game Among Smart Microgrids , 2016, IEEE Transactions on Industrial Electronics.

[102]  Enrico Zio,et al.  A reinforcement learning framework for optimal operation and maintenance of power grids , 2019, Applied Energy.

[103]  Junwei Cao,et al.  Optimal energy management strategies for energy Internet via deep reinforcement learning approach , 2019, Applied Energy.

[104]  Bin Liu,et al.  Fast learning optimiser for real-time optimal energy management of a grid-connected microgrid , 2018 .

[105]  Zhuoran Yang,et al.  A Theoretical Analysis of Deep Q-Learning , 2019, L4DC.

[106]  D. Bertsekas.  Reinforcement Learning and Optimal ControlA Selective Overview , 2018 .

[107]  Renke Huang,et al.  Adaptive Power System Emergency Control Using Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[108]  Qian Ai,et al.  A Novel Data-Driven Situation Awareness Approach for Future Grids—Using Large Random Matrices for Big Data Modeling , 2016, IEEE Access.

[109]  Mohammad Reza Salehizadeh,et al.  Application of fuzzy Q-learning for electricity market modeling by considering renewable power penetration , 2016 .

[110]  Yuxi Li,et al.  Deep Reinforcement Learning , 2018, Reinforcement Learning for Cyber-Physical Systems.

[111]  Youyong Kong,et al.  Deep Direct Reinforcement Learning for Financial Signal Representation and Trading , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[112]  Elena Mocanu,et al.  Machine learning applied to smart grids , 2017 .