Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning

Abstract Whole building energy model (BEM) is a physics-based modeling method for building energy simulation. It has been widely used in the building industry for code compliance, building design optimization, retrofit analysis, and other uses. Recent research also indicates its strong potential for the control of heating, ventilation and air-conditioning (HVAC) systems. However, its high-order nature and slow computational speed limit its practical application in real-time HVAC optimal control. Therefore, this study proposes a practical control framework (named BEM-DRL) that is based on deep reinforcement learning. The framework is implemented and assessed in a novel radiant heating system in an existing office building as a case study. The complete implementation process is presented in this study, including: building energy modeling for the novel heating system, multi-objective BEM calibration using the Bayesian method and the Genetic Algorithm, deep reinforcement learning training and simulation results evaluation, and control deployment. By analyzing the real-life control deployment data, it is found that BEM-DRL achieves 16.7% heating demand reduction with more than 95% probability compared to the old rule-based control. However, the framework still faces the practical challenges including building energy modeling of novel HVAC systems and multi-objective model calibration. Systematic study is also needed for the design of deep reinforcement learning training to provide a guideline for practitioners.

[1]  Lukas Ferkl,et al.  Model-based energy efficient control applied to an office building , 2014 .

[2]  Gerardo Maria Mauro,et al.  Simulation-based model predictive control by the multi-objective optimization of building energy performance and thermal comfort , 2016 .

[3]  Max D. Morris,et al.  Factorial sampling plans for preliminary computational experiments , 1991 .

[4]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[5]  David E. Claridge,et al.  Ambient-temperature regression analysis for estimating retrofit savings in commercial buildings , 1998 .

[6]  Wei Liang,et al.  MPC control for improving energy efficiency of a building air handler for multi-zone VAVs , 2015 .

[7]  Michael Wetter,et al.  Building Controls Virtual Test Bed , 2008 .

[8]  Biao Huang,et al.  A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems , 2017 .

[9]  Zhen Yu,et al.  Online tuning of a supervisory fuzzy controller for low-energy building system using reinforcement learning , 2010 .

[10]  Hossein Afshari,et al.  Field tests of an adaptive, model-predictive heating controller for residential buildings , 2015 .

[11]  Jie Zhao,et al.  EnergyPlus model-based predictive control within design–build–operate energy information modelling infrastructure , 2015 .

[12]  Josh Wall,et al.  Trial results from a model predictive control and optimisation system for commercial building HVAC , 2014 .

[13]  Cheol-Yong Jang,et al.  Development of a model predictive control framework through real-time building energy management system data , 2015 .

[14]  Adrian Chong,et al.  Guidelines for the Bayesian calibration of building energy models , 2018, Energy and Buildings.

[15]  Xiao Chen,et al.  Occupant feedback based model predictive control for thermal comfort and energy optimization: A chamber experimental evaluation , 2016 .

[16]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis , 2006 .

[17]  Dongbin Zhao,et al.  Thermal comfort control based on MEC algorithm for HVAC systems , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[18]  Clayton T. Morrison,et al.  Model Predictive Prior Reinforcement Learning for a Heat Pump Thermostat , 2016 .

[19]  D. Kolokotsa,et al.  Reinforcement learning for energy conservation and comfort in buildings , 2007 .

[20]  José R. Vázquez-Canteli,et al.  Balancing comfort and energy consumption of a heat pump using batch reinforcement learning with fitted Q-iteration , 2017 .

[21]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[22]  José R. Vázquez-Canteli,et al.  Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities , 2019, Sustainable Cities and Society.

[23]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[24]  Tianshu Wei,et al.  Deep reinforcement learning for building HVAC control , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[25]  Talal Rahwan,et al.  Automatic HVAC Control with Real-time Occupancy Recognition and Simulation-guided Model Predictive Control in Low-cost Embedded System , 2017, ArXiv.

[26]  Khee Poh Lam,et al.  Practical implementation and evaluation of deep reinforcement learning control for a radiant heating system , 2018, BuildSys@SenSys.

[27]  Simeng Liu,et al.  Evaluation of reinforcement learning for optimal control of building active and passive thermal storage inventory , 2007 .

[28]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[29]  Kalyan Veeramachaneni,et al.  Using reinforcement learning to optimize occupant comfort and energy usage in HVAC systems , 2014, J. Ambient Intell. Smart Environ..

[30]  Giuseppe Tommaso Costanzo,et al.  Experimental analysis of data-driven control for a building heating system , 2015, ArXiv.

[31]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[32]  Yonggang Wen,et al.  Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning , 2017, IEEE Transactions on Cybernetics.

[33]  Mahdi Shahbakhti,et al.  Optimal exergy control of building HVAC system , 2015 .

[34]  Jianjun Hu,et al.  System identification and model-predictive control of office buildings with integrated photovoltaic-thermal collectors, radiant floor heating and active thermal storage , 2015 .

[35]  Gordon Lightbody,et al.  Prioritised objectives for model predictive control of building heating systems , 2017 .

[36]  Nanpeng Yu,et al.  Energy Efficient Building HVAC Control Algorithm with Real-time Occupancy Prediction , 2017 .

[37]  Balaji Rajagopalan,et al.  Model-predictive control of mixed-mode buildings with rule extraction , 2011 .

[38]  S. Joe Qin,et al.  Application of economic MPC to the energy and demand minimization of a commercial building , 2014 .

[39]  Martins Miezis,et al.  Predictive Control of a Building Heating System , 2017 .

[40]  Ronnie Belmans,et al.  Learning Agent for a Heat-Pump Thermostat With a Set-Back Strategy Using Model-Free Reinforcement Learning , 2015, ArXiv.

[41]  Panos J. Antsaklis,et al.  Model-based predictive control for building energy management: Part II – Experimental validations , 2017 .

[42]  Junjing Yang,et al.  Bayesian calibration of building energy models with large datasets , 2017 .

[43]  Lei Chen,et al.  A new model predictive control scheme for energy and cost savings in commercial buildings: An airport terminal building case study , 2015 .

[44]  Anton Kummert,et al.  Model Predictive Control for Hydronic Heating Systems in Residential Buildings , 2017 .

[45]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[46]  Song Chao,et al.  Continuous-time Bayesian calibration of energy models using BIM and energy data , 2019, Energy and Buildings.

[47]  Francesco Borrelli,et al.  Stochastic Model Predictive Control for Building HVAC Systems: Complexity and Conservatism , 2015, IEEE Transactions on Control Systems Technology.

[48]  P. O. Fanger,et al.  Thermal comfort: analysis and applications in environmental engineering, , 1972 .

[49]  Lieve Helsen,et al.  Practical implementation and evaluation of model predictive control for an office building in Brussels , 2016 .

[50]  Leslie K. Norford,et al.  Optimal control of HVAC and window systems for natural ventilation through reinforcement learning , 2018, Energy and Buildings.

[51]  Ardeshir Mahdavi.  Simulation-based control of building systems operation , 2001 .

[52]  Lei Yang,et al.  Reinforcement learning for optimal control of low exergy buildings , 2015 .

[53]  Daniel Urieli,et al.  A learning agent for heat-pump thermostat control , 2013, AAMAS.

[54]  Michael C. Mozer,et al.  The Neurothermostat: Predictive Optimal Control of Residential Heating Systems , 1996, NIPS.

[55]  Johan Driesen,et al.  Deep Reinforcement Learning for Optimal Control of Space Heating , 2018, ArXiv.

[56]  Siliang Lu,et al.  A DEEP REINFORCEMENT LEARNING APPROACH TO USINGWHOLE BUILDING ENERGYMODEL FOR HVAC OPTIMAL CONTROL , 2018 .

[57]  Martin Kozek,et al.  Implementation of cooperative Fuzzy model predictive control for an energy-efficient office building , 2018 .