Modified deep learning and reinforcement learning for an incentive-based demand response model

Abstract Incentive-based demand response (DR) program can induce end users (EUs) to reduce electricity demand during peak period through rewards. In this study, an incentive-based DR program with modified deep learning and reinforcement learning is proposed. A modified deep learning model based on recurrent neural network (MDL-RNN) was first proposed to identify the future uncertainties of environment by forecasting day-ahead wholesale electricity price, photovoltaic (PV) power output, and power load. Then, reinforcement learning (RL) was utilized to explore the optimal incentive rates at each hour which can maximize the profits of both energy service providers (ESPs) and EUs. The results showed that the proposed modified deep learning model can achieve more accurate forecasting results compared with some other methods. It can support the development of incentive-based DR programs under uncertain environment. Meanwhile, the optimized incentive rate can increase the total profits of ESPs and EUs while reducing the peak electricity demand. A short-term DR program was developed for peak electricity demand period, and the experimental results show that peak electricity demand can be reduced by 17%. This contributes to mitigating the supply-demand imbalance and enhancing power system security.

[1]  Antonello Monti,et al.  Hierarchical Distributed Robust Optimization for Demand Response Services , 2018, IEEE Transactions on Smart Grid.

[2]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[3]  Seung Ho Hong,et al.  Demand Response for Home Energy Management Using Reinforcement Learning and Artificial Neural Network , 2019, IEEE Transactions on Smart Grid.

[4]  Mengmeng Yu,et al.  Incentive-based demand response considering hierarchical electricity market: A Stackelberg game approach , 2017 .

[5]  Xudong Zhang,et al.  Deep reinforcement learning based energy management for a hybrid electric vehicle , 2020, Energy.

[6]  Chuntian Cheng,et al.  Annual Streamflow Time Series Prediction Using Extreme Learning Machine Based on Gravitational Search Algorithm and Variational Mode Decomposition , 2020 .

[7]  K. K. Sahu,et al.  Normalization: A Preprocessing Stage , 2015, ArXiv.

[8]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[9]  Tianqi Hong,et al.  Dynamic Demand Response Using Customer Coupons Considering Multiple Load Aggregators to Simultaneously Achieve Efficiency and Fairness , 2018, IEEE Transactions on Smart Grid.

[10]  Mehdi Rahmani-andebili,et al.  Modeling nonlinear incentive-based and price-based demand response programs and implementing on real power markets , 2016 .

[11]  Ahmad Ghasemi,et al.  A hybrid price-based demand response program for the residential micro-grid , 2019, Energy.

[12]  Ning Zhang,et al.  Probabilistic individual load forecasting using pinball loss guided LSTM , 2019, Applied Energy.

[13]  Dipti Srinivasan,et al.  Game-Theory based dynamic pricing strategies for demand side management in smart grids , 2017 .

[14]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[15]  Fangxing Li,et al.  Coupon-Based Demand Response Considering Wind Power Uncertainty: A Strategic Bidding Model for Load Serving Entities , 2016, IEEE Transactions on Power Systems.

[16]  Hairong Qi,et al.  Evaluation of residential customer elasticity for incentive based demand response programs , 2018 .

[17]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[18]  Siobhán Clarke,et al.  Prediction-Based Multi-Agent Reinforcement Learning in Inherently Non-Stationary Environments , 2017, ACM Trans. Auton. Adapt. Syst..

[19]  Victor C. M. Leung,et al.  Energy Management in Smart Cities Based on Internet of Things: Peak Demand Reduction and Energy Savings , 2017, Sensors.

[20]  Jun Wang,et al.  Model Predictive Control of Nonlinear Systems With Unmodeled Dynamics Based on Feedforward and Recurrent Neural Networks , 2012, IEEE Transactions on Industrial Informatics.

[21]  João P. S. Catalão,et al.  Novel Incentive Mechanism for End-Users Enrolled in DLC-Based Demand Response Programs Within Stochastic Planning Context , 2019, IEEE Transactions on Industrial Electronics.

[22]  Weihao Hu,et al.  A Heuristic Planning Reinforcement Learning-Based Energy Management for Power-Split Plug-in Hybrid Electric Vehicles , 2019, IEEE Transactions on Industrial Informatics.

[23]  Seung Ho Hong,et al.  A real-time decision model for industrial load management in a smart grid , 2016 .

[24]  Lei Yang,et al.  Learning-Based Demand Response for Privacy-Preserving Users , 2019, IEEE Transactions on Industrial Informatics.

[25]  Sergei Vassilvitskii,et al.  Fast greedy algorithms in mapreduce and streaming , 2013, SPAA.

[26]  Seung Ho Hong,et al.  A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach , 2018, Applied Energy.

[27]  Behnam Mohammadi-Ivatloo,et al.  Optimal battery technology selection and incentive-based demand response program utilization for reliability improvement of an insular microgrid , 2019, Energy.

[28]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[29]  Chuntian Cheng,et al.  Peak shaving operation of hydro-thermal-nuclear plants serving multiple power grids by linear programming , 2017 .

[30]  Seung Ho Hong,et al.  Real-Time Demand Bidding for Energy Management in Discrete Manufacturing Facilities , 2017, IEEE Transactions on Industrial Electronics.

[31]  Ailin Asadinejad,et al.  Optimal use of incentive and price based demand response to reduce costs and price volatility , 2017 .

[32]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[33]  Giuseppe Tommaso Costanzo,et al.  Experimental analysis of data-driven control for a building heating system , 2015, ArXiv.

[34]  Peggy Seriès,et al.  Reward-Based Learning, Model-Based and Model-Free , 2014, Encyclopedia of Computational Neuroscience.

[35]  Wen-jing Niu,et al.  Monthly runoff time series prediction by variational mode decomposition and support vector machine based on quantum-behaved particle swarm optimization , 2020 .

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[38]  Seung Ho Hong,et al.  Incentive-based demand response for smart grid with reinforcement learning and deep neural network , 2019, Applied Energy.

[39]  Ong Hang See,et al.  A review of residential demand response of smart grid , 2016 .

[40]  Behnam Mohammadi-Ivatloo,et al.  An improved incentive-based demand response program in day-ahead and intra-day electricity markets , 2018, Energy.

[41]  Muhammad Babar,et al.  Online scheduling of plug-in vehicles in dynamic pricing schemes , 2016 .

[42]  Pieter Abbeel,et al.  Equivalence Between Policy Gradients and Soft Q-Learning , 2017, ArXiv.

[43]  Vivek Srikumar,et al.  Predicting electricity consumption for commercial and residential buildings using deep recurrent neural networks , 2018 .

[44]  Shanlin Yang,et al.  A deep learning model for short-term power load and probability density forecasting , 2018, Energy.

[45]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[46]  Mehdi Rahmani-andebili Nonlinear demand response programs for residential customers with nonlinear behavioral models , 2016 .

[47]  Seung Ho Hong,et al.  Incentive-based demand response approach for aggregated demand side participation , 2016, 2016 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[48]  Sen Wang,et al.  An effective three-stage hybrid optimization method for source-network-load power generation of cascade hydropower reservoirs serving multiple interconnected power grids , 2020 .

[49]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[50]  Zechun Hu,et al.  A review on price-driven residential demand response , 2018, Renewable and Sustainable Energy Reviews.

[51]  Mehdi Rahmani-Andebili,et al.  Energy management of end users modeling their reaction from a GENCO's point of view , 2017, 2017 International Conference on Computing, Networking and Communications (ICNC).

[52]  Zita Vale,et al.  Incentive-based demand response programs designed by asset-light retail electricity providers for the day-ahead market , 2015 .

[53]  Jing Peng,et al.  Incremental multi-step Q-learning , 1994, Machine-mediated learning.

[54]  Mengmeng Yu,et al.  Supply–demand balancing for power management in smart grid: A Stackelberg game approach , 2016 .

[55]  Tingwen Huang,et al.  Reinforcement Learning in Energy Trading Game Among Smart Microgrids , 2016, IEEE Transactions on Industrial Electronics.

[56]  Lei Yan,et al.  Interactive Dispatch Modes and Bidding Strategy of Multiple Virtual Power Plants Based on Demand Response and Game Theory , 2016, IEEE Transactions on Smart Grid.