Real-Time Autonomous Residential Demand Response Management Based on Twin Delayed Deep Deterministic Policy Gradient Learning

With the roll-out of smart meters and the increasing prevalence of distributed energy resources (DERs) at the residential level, end-users rely on home energy management systems (HEMSs) that can harness real-time data and employ artificial intelligence techniques to optimally manage the operation of different DERs, which are targeted toward minimizing the end-user’s energy bill. In this respect, the performance of the conventional model-based demand response (DR) management approach may deteriorate due to the inaccuracy of the employed DER operating models and the probabilistic modeling of uncertain parameters. To overcome the above drawbacks, this paper develops a novel real-time DR management strategy for a residential household based on the twin delayed deep deterministic policy gradient (TD3) learning approach. This approach is model-free, and thus does not rely on knowledge of the distribution of uncertainties or the operating models and parameters of the DERs. It also enables learning of neural-network-based and fine-grained DR management policies in a multi-dimensional action space by exploiting high-dimensional sensory data that encapsulate the uncertainties associated with the renewable generation, appliances’ operating states, utility prices, and outdoor temperature. The proposed method is applied to the energy management problem for a household with a portfolio of the most prominent types of DERs. Case studies involving a real-world scenario are used to validate the superior performance of the proposed method in reducing the household’s energy costs while coping with the multi-source uncertainties through comprehensive comparisons with the state-of-the-art deep reinforcement learning (DRL) methods.

[1]  Antonio Alonso Ayuso,et al.  Introduction to Stochastic Programming , 2009 .

[2]  Miadreza Shafie-Khah,et al.  A Stochastic Home Energy Management System Considering Satisfaction Cost and Response Fatigue , 2018, IEEE Transactions on Industrial Informatics.

[3]  Damien Ernst,et al.  Deep Reinforcement Learning Solutions for Energy Microgrids Management , 2016 .

[4]  E. A. Jasmin,et al.  Residential Load Scheduling With Renewable Generation in the Smart Grid: A Reinforcement Learning Approach , 2019, IEEE Systems Journal.

[5]  Goran Strbac,et al.  Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning , 2020, IEEE Transactions on Smart Grid.

[6]  L. Jiang,et al.  Energy Consumption Scheduling of HVAC Considering Weather Forecast Error Through the Distributionally Robust Approach , 2018, IEEE Transactions on Industrial Informatics.

[7]  Factoring flexible demand non-convexities in electricity markets , 2015, 2016 IEEE Power and Energy Society General Meeting (PESGM).

[8]  Steven R. Weller,et al.  Residential load and rooftop PV generation: an Australian distribution network dataset , 2017 .

[9]  Anna Helena Reali Costa,et al.  Intelligent Decision-Making for Smart Home Energy Management , 2014, Journal of Intelligent & Robotic Systems.

[10]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[11]  Bart De Schutter,et al.  Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[12]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[13]  Tianshu Wei,et al.  Deep reinforcement learning for building HVAC control , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[14]  Kankar Bhattacharya,et al.  Optimal Operation of Residential Energy Hubs in Smart Grids , 2012, IEEE Transactions on Smart Grid.

[15]  Lin Jiang,et al.  A Robust Optimization Approach for Demand Side Scheduling Considering Uncertainty of Manually Operated Appliances , 2018, IEEE Transactions on Smart Grid.

[16]  Hao Wang,et al.  Energy Storage Arbitrage in Real-Time Markets via Reinforcement Learning , 2017, 2018 IEEE Power & Energy Society General Meeting (PESGM).

[17]  Ikhlaq Sidhu,et al.  Autonomous Household Energy Management Using Deep Reinforcement Learning , 2019, 2019 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC).

[18]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[19]  Lei Wang,et al.  Chance Constrained Optimization in a Home Energy Management System , 2018, IEEE Transactions on Smart Grid.

[20]  Hado van Hasselt,et al.  Double Q-learning , 2010, NIPS.

[21]  Huaguang Zhang,et al.  Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning , 2019, Energies.

[22]  Tao Jiang,et al.  Online Energy Management for a Sustainable Smart Home With an HVAC Load and Random Occupancy , 2017, IEEE Transactions on Smart Grid.

[23]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[24]  Goran Strbac,et al.  Nonlinear and Randomized Pricing for Distributed Management of Flexible Loads , 2016, IEEE Transactions on Smart Grid.

[25]  Tao Chen,et al.  Local Energy Trading Behavior Modeling With Deep Reinforcement Learning , 2018, IEEE Access.

[26]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[27]  Goran Strbac,et al.  Investigating the impacts of price‐taking and price‐making energy storage in electricity markets through an equilibrium programming model , 2018, IET Generation, Transmission & Distribution.

[28]  Feifei Gao,et al.  Efficient and Autonomous Energy Management Techniques for the Future Smart Homes , 2017, IEEE Transactions on Smart Grid.

[29]  Zhi Chen,et al.  Real-Time Price-Based Demand Response Management for Residential Appliances via Stochastic Optimization and Robust Optimization , 2012, IEEE Transactions on Smart Grid.

[30]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[31]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[32]  Long He,et al.  Stochastic Control for Smart Grid Users With Flexible Demand , 2013, IEEE Transactions on Smart Grid.

[33]  Zheng Wen,et al.  Optimal Demand Response Using Device-Based Reinforcement Learning , 2014, IEEE Transactions on Smart Grid.

[34]  R. Belmans,et al.  Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice , 2015, IEEE Transactions on Smart Grid.

[35]  Doina Precup,et al.  Optimizing Home Energy Management and Electric Vehicle Charging with Reinforcement Learning , 2018 .

[36]  Elaheh Mashhour,et al.  A comprehensive model for self-scheduling an energy hub to supply cooling, heating and electrical demands of a building , 2016 .

[37]  Jeff Smith,et al.  Distributed Energy Resources Takes Center Stage: A Renewed Spotlight on the Distribution Planning Process , 2018, IEEE Power and Energy Magazine.

[38]  Mahmud Fotuhi-Firuzabad,et al.  Load management in a residential energy hub with renewable distributed energy resources , 2015 .

[39]  Herke van Hoof,et al.  Addressing Function Approximation Error in Actor-Critic Methods , 2018, ICML.

[40]  Hyuk Lim,et al.  Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings , 2018, Energies.

[41]  Iain MacGill,et al.  Coordinated Scheduling of Residential Distributed Energy Resources to Optimize Smart Home Energy Services , 2010, IEEE Transactions on Smart Grid.

[42]  Constantine Caramanis,et al.  Theory and Applications of Robust Optimization , 2010, SIAM Rev..

[43]  Shahin Nazarian,et al.  Reinforcement learning-based control of residential energy storage systems for electric bill minimization , 2015, 2015 12th Annual IEEE Consumer Communications and Networking Conference (CCNC).

[44]  Haibo He,et al.  Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning , 2019, IEEE Transactions on Smart Grid.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Haibo He,et al.  Residential Energy Management with Deep Reinforcement Learning , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).