Trajectory Tracking With an Aggregation of Domestic Hot Water Heaters: Combining Model-Based and Model-Free Control in a Commercial Deployment

Scalable demand response of residential electric loads has been a timely research topic in recent years. The commercial coming of age of residential demand response requires a scalable control architecture that is both efficient and practical to use. This paper presents such a strategy for domestic hot water heaters and presents a commercial proof-of-concept deployment. The strategy combines state-of-the-art in aggregate-and-dispatch with a novel dispatch strategy leveraging recent developments in reinforcement learning and is tested in a hardware-in-the-loop experiment environment. The results are promising and present how model-based and model-free control strategies can be merged to obtain a mature and commercially viable control strategy for residential demand response.

[1]  Johan Driesen,et al.  Multiagent Charging of Electric Vehicles Respecting Distribution Transformer Loading and Voltage Limits , 2014, IEEE Transactions on Smart Grid.

[2]  Geert Deconinck,et al.  Cluster Control of Heterogeneous Thermostatically Controlled Loads Using Tracer Devices , 2017, IEEE Transactions on Smart Grid.

[3]  Tyrone L. Vincent,et al.  Aggregate Flexibility of Thermostatically Controlled Loads , 2015, IEEE Transactions on Power Systems.

[4]  Hussain Kazmi,et al.  Demonstrating model-based reinforcement learning for energy efficiency and demand response using hot water vessels in net-zero energy buildings , 2016, 2016 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe).

[5]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[6]  Junjie HU,et al.  Transactive control: a framework for operating power systems characterized by high penetration of distributed energy resources , 2017 .

[7]  Ronnie Belmans,et al.  Demand response of a heterogeneous cluster of electric water heaters using batch reinforcement learning , 2014, 2014 Power Systems Computation Conference.

[8]  Zhu Han,et al.  Multi-block ADMM for big data optimization in smart grid , 2015, 2015 International Conference on Computing, Networking and Communications (ICNC).

[9]  Bart De Schutter,et al.  Reinforcement Learning and Dynamic Programming Using Function Approximators , 2010 .

[10]  Colin Neil Jones,et al.  Guaranteeing input tracking for constrained systems: Theory and application to demand response , 2015, 2015 American Control Conference (ACC).

[11]  Ian A. Hiskens,et al.  Achieving Controllability of Electric Loads , 2011, Proceedings of the IEEE.

[12]  Yang Shi,et al.  Model Predictive Control of Aggregated Heterogeneous Second-Order Thermostatically Controlled Loads for Ancillary Services , 2016, IEEE Transactions on Power Systems.

[13]  Hado van Hasselt,et al.  Double Q-learning , 2010, NIPS.

[14]  Koen Vanthournout,et al.  A Smart Domestic Hot Water Buffer , 2012, IEEE Transactions on Smart Grid.

[15]  Tom Holvoet,et al.  A Scalable Three-Step Approach for Demand Side Management of Plug-in Hybrid Vehicles , 2013, IEEE Transactions on Smart Grid.

[16]  Yang Shi,et al.  Distributed MPC of Aggregated Heterogeneous Thermostatically Controlled Loads in Smart Grid , 2016, IEEE Transactions on Industrial Electronics.

[17]  Evangelos Vrettos,et al.  Load frequency control by aggregations of thermally stratified electric water heaters , 2012, 2012 3rd IEEE PES Innovative Smart Grid Technologies Europe (ISGT Europe).

[18]  J. K. Kok,et al.  PowerMatcher: multiagent control in the electricity infrastructure , 2005, AAMAS '05.

[19]  Bart De Schutter,et al.  Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[20]  Duncan S. Callaway,et al.  Location, Location, Location: The Variable Value of Renewable Energy and Demand-Side Efficiency Resources , 2018, Journal of the Association of Environmental and Resource Economists.

[21]  Liuchen Chang,et al.  A novel domestic electric water heater model for a multi-objective demand side management program , 2010 .

[22]  John Lygeros,et al.  Energy arbitrage with thermostatically controlled loads , 2013, 2013 European Control Conference (ECC).

[23]  Kai Heussen,et al.  Energy storage in power system operation: The power nodes modeling framework , 2010, 2010 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT Europe).

[24]  Peter Vrancx,et al.  Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control , 2016, IEEE Transactions on Smart Grid.

[25]  Georgios B. Giannakis,et al.  Residential Load Control: Distributed Scheduling and Convergence With Lost AMI Messages , 2012, IEEE Transactions on Smart Grid.

[26]  Ronnie Belmans,et al.  A Flexible Stochastic Optimization Method for Wind Power Balancing With PHEVs , 2014, IEEE Transactions on Smart Grid.

[27]  Marco Levorato,et al.  Residential Demand Response Using Reinforcement Learning , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[28]  Johan Driesen,et al.  Deep Reinforcement Learning based Optimal Control of Hot Water Systems , 2018, ArXiv.

[29]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[30]  Zhiwei Xu,et al.  Hierarchical Coordination of Heterogeneous Flexible Loads , 2016, IEEE Transactions on Power Systems.

[31]  R. Belmans,et al.  Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice , 2015, IEEE Transactions on Smart Grid.

[32]  Pierre Geurts,et al.  Extremely randomized trees , 2006, Machine Learning.

[33]  Duncan S. Callaway,et al.  State Estimation and Control of Electric Loads to Manage Real-Time Energy Imbalance , 2013, IEEE Transactions on Power Systems.

[34]  Wen-Yeau Chang,et al.  A Literature Review of Wind Forecasting Methods , 2014 .

[35]  Dirk Vanhoudt,et al.  Model-Free Control of Thermostatically Controlled Loads Connected to a District Heating Network , 2017, ArXiv.

[36]  Johanna L. Mathieu,et al.  Modeling and Control of Aggregated Heterogeneous Thermostatically Controlled Loads for Ancillary Services , 2011 .

[37]  Evangelos Vrettos,et al.  Combined Load Frequency Control and active distribution network management with Thermostatically Controlled Loads , 2013, 2013 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[38]  Dilek Z. Hakkani-Tür,et al.  Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning , 2017, ArXiv.

[39]  R. Bellman.  Dynamic programming. , 1957, Science.

[40]  Giuseppe Tommaso Costanzo,et al.  Experimental analysis of data-driven control for a building heating system , 2015, ArXiv.