Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating

Abstract Applications of electric heating, which can improve carbon emission reduction and renewable energy utilization, have brought new challenges to the safe operation of energy systems around the world. Regenerative electric heating with load aggregators and demand response is an effective means to mitigate the wind curtailment and grid operational risks caused by electric heating. However, there is still a lack of models related to demand response, which results in participants not being able to obtain maximum benefits through dynamic subsidy prices. This study uses the Weber–Fechner law and a clustering algorithm to construct quantitative response characteristics models. The deep Q network was used to build a dynamic subsidy price generation framework for load aggregators. Through simulation analysis based on the evolutionary game model of a project in a rural area in Tianjin, China, the following conclusions were drawn: compared with the benchmark model, regenerative electric heating users can save up to 8.7% of costs, power grid companies can save 56.6% of their investment, and wind power plants can increase wind power consumption by 17.6%. The framework proposed in this study considers user behavior quantification of demand response participants and the differences among users. Therefore, the framework can provide a more reasonable, applicable, and intelligent system for regenerative electric heating.

[1]  Jianzhong Wu,et al.  Optimal scheduling of aggregated thermostatically controlled loads with renewable generation in the intraday electricity market , 2017 .

[2]  Amjad Anvari-Moghaddam,et al.  Optimal Behavior of a Hybrid Power Producer in Day-Ahead and Intraday Markets: A Bi-Objective CVaR-Based Approach , 2021, IEEE Transactions on Sustainable Energy.

[3]  Duncan S. Callaway.  Tapping the energy storage potential in electric loads to deliver load following and regulation, with application to wind energy , 2009 .

[4]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[5]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[6]  T. Dragičević,et al.  Bidding strategy for trading wind energy and purchasing reserve of wind power producer – A DRL based approach , 2020 .

[7]  Shahin Pourbahrami,et al.  A survey of neighborhood construction algorithms for clustering and classifying data points , 2020, Comput. Sci. Rev..

[8]  S.E. Widergren,et al.  Modeling uncertainties in aggregated thermostatically controlled loads using a State queueing model , 2005, IEEE Transactions on Power Systems.

[9]  Pranab J. Baruah,et al.  Uncertainties in future energy demand in UK residential heating , 2015 .

[10]  Wei Wang,et al.  Multi-objective electro-thermal coupling scheduling model for a hybrid energy system comprising wind power plant, conventional gas turbine, and regenerative electric boiler, considering uncertainty and demand response , 2019, Journal of Cleaner Production.

[11]  Ling-Ling Li,et al.  Improving power quality efficient in demand response: Aggregated heating, ventilation and air-conditioning systems , 2020 .

[12]  F. Johnsson,et al.  Impacts of demand response from buildings and centralized thermal energy storage on district heating systems , 2021 .

[13]  Shan Wang,et al.  Optimization modeling method for coal-to-electricity heating load considering differential decisions , 2019 .

[14]  Mao Yang,et al.  Day-ahead wind power forecasting based on the clustering of equivalent power curves , 2021, Energy.

[15]  Sergio Cesare Masin,et al.  Early alternative derivations of Fechner's law. , 2009, Journal of the history of the behavioral sciences.

[16]  Matti Lehtonen,et al.  A control framework for the utilization of heating load flexibility in a day-ahead market , 2017 .

[17]  Eric G. O'Rear,et al.  Gas vs electric: Heating system fuel source implications on low-energy single-family dwelling sustainability performance , 2019, Journal of Building Engineering.

[18]  Di Cao,et al.  Deep reinforcement learning–based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy , 2019, Energy Conversion and Management.

[19]  Yuping Lu,et al.  Optimal design and operation of multi-energy system with load aggregator considering nodal energy prices , 2019, Applied Energy.

[20]  Marko Aunedi,et al.  Modelling of national and local interactions between heat and electricity networks in low-carbon energy systems , 2020 .

[21]  Stanislas Dehaene,et al.  The neural basis of the Weber–Fechner law: a logarithmic mental number line , 2003, Trends in Cognitive Sciences.

[22]  Kai Fang,et al.  Will China peak its energy-related carbon emissions by 2030? Lessons from 30 Chinese provinces , 2019 .

[23]  Xiaodi Wang,et al.  Thermally controllable demand response with multiple load aggregators , 2020 .

[24]  Xue Li,et al.  Collaborative scheduling and flexibility assessment of integrated electricity and district heating systems utilizing thermal inertia of district heating network and aggregated buildings , 2020 .

[25]  Z. Tan,et al.  Feasible electricity price calculation and environmental benefits analysis of the regional nighttime wind power utilization in electric heating in Beijing , 2019, Journal of Cleaner Production.

[26]  Albert Y. Zomaya,et al.  Reinforcement learning in sustainable energy and electric systems: a survey , 2020, Annu. Rev. Control..

[27]  Yunfei Mu,et al.  Bi-Level Optimization Framework for Buildings to Heating Grid Integration in Integrated Community Energy Systems , 2021, IEEE Transactions on Sustainable Energy.

[28]  L. Yang,et al.  Illustrating Economic Reform and Development of China During the 13th Five-Year Plan Period , 2016 .

[29]  Jianzhong Wu,et al.  k-means based load estimation of domestic smart meter measurements , 2017 .

[30]  Shengwei Mei,et al.  Coal or electricity? An evolutionary game approach to investigate fuel choices of urban heat supply systems , 2019, Energy.

[31]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[32]  Yang Yang,et al.  Image quality assessment based on the space similarity decomposition model , 2016, Signal Process..

[33]  Shuba V. Raghavan,et al.  Translating climate change and heating system electrification impacts on building energy use to future greenhouse gas emissions and electric grid capacity requirements in California , 2018, Applied Energy.

[34]  Niu Jianhui,et al.  ‘Coal-to-electricity’ project is ongoing in north China , 2020 .

[35]  Xiaonan Wang,et al.  Long-term economic planning of combined cooling heating and power systems considering energy storage and demand response , 2020 .

[36]  Hong Qu,et al.  A deep reinforcement learning based long-term recommender system , 2021, Knowl. Based Syst..

[37]  Xi Lu,et al.  Decomposing driving factors for wind curtailment under economic new normal in China , 2018 .

[38]  Radoslaw Martin Cichy,et al.  Deep Neural Networks as Scientific Models , 2019, Trends in Cognitive Sciences.

[39]  Neven Duić,et al.  Impact of wind penetration in electricity markets on optimal power-to-heat capacities in a local district heating system , 2020 .

[40]  Taiki Takahashi,et al.  Time-estimation error following Weber-Fechner law may explain subadditive time-discounting. , 2006, Medical hypotheses.

[41]  Hongwen He,et al.  Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus , 2018, Applied Energy.

[42]  Parameswaran Kamalaruban,et al.  Applications of reinforcement learning in energy systems , 2021, Renewable and Sustainable Energy Reviews.

[43]  Linquan Bai,et al.  Online pricing of demand response based on long short-term memory and reinforcement learning , 2020 .

[44]  William D'haeseleer,et al.  Integrated modeling of active demand response with electric heating systems coupled to thermal energy storage systems , 2015 .

[45]  Filip Johnsson,et al.  Impact of electricity price fluctuations on the operation of district heating systems: A case study of district heating in Göteborg, Sweden , 2017 .

[46]  Julio Usaola,et al.  An optimal day-ahead load scheduling approach based on the flexibility of aggregate demands , 2017 .

[47]  Paweł Cichosz,et al.  Real-time energy purchase optimization for a storage-integrated photovoltaic system by deep reinforcement learning , 2021 .

[48]  Shiyi Tian,et al.  A new sensory sweetness definition and sweetness conversion method of five natural sugars, based on the Weber-Fechner Law. , 2019, Food chemistry.

[49]  Luis Lopez,et al.  EU carbon emissions by multinational enterprises under control-based accounting , 2020 .

[50]  Weijun Wang,et al.  Study on substitutable value of electric heating instead of coal heating in northern China under carbon constraints , 2020 .

[51]  Y. Nakanishi,et al.  Demand response model based on improved Pareto optimum considering seasonal electricity prices for Dongfushan Island , 2021 .