Practical implementation and evaluation of deep reinforcement learning control for a radiant heating system

Deep reinforcement learning (DRL) has become a popular optimal control method in recent years. This is mainly because DRL has the potential to solve the optimal control problems with complex process dynamics, such as the optimal control for heating, ventilation, and air-conditioning (HVAC) systems. However, DRL control for HVAC systems has not been well studied. There is limited research on the real-life implementation and evaluation of this method. This study implements and deploys a DRL control method for a radiant heating system in a real-life office building for energy efficiency. A physics-based model for the heating system is first created and then calibrated using the measured building operation data. After that, the model is used as a simulator to train the DRL agent. The trained agent is then deployed in the actual heating system, and a smartphone App is used to let the occupants submit their thermal preferences to the DRL agent. It is found the DRL control method can save 16.6% to 18.2% heating demand compared to the old rule-based control logic over the three-month deployment period. However, several limitations of this study are found, such as the low participation rate of the App-based thermal preference feedback system, inefficient DRL training, and the requirement for a large amount of building data.

[1]  Biao Huang,et al.  A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems , 2017 .

[2]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[3]  Giuseppe Tommaso Costanzo,et al.  Experimental analysis of data-driven control for a building heating system , 2015, ArXiv.

[4]  Simeng Liu,et al.  Evaluation of reinforcement learning for optimal control of building active and passive thermal storage inventory , 2007 .

[5]  Jie Zhao,et al.  EnergyPlus model-based predictive control within design–build–operate energy information modelling infrastructure , 2015 .

[6]  Mahdi Shahbakhti,et al.  Optimal exergy control of building HVAC system , 2015 .

[7]  Lukas Ferkl,et al.  Model-based energy efficient control applied to an office building , 2014 .

[8]  Michael Wetter,et al.  Building Controls Virtual Test Bed , 2008 .

[9]  Juhi Ranjan,et al.  ThermalSense: determining dynamic thermal comfort preferences using thermographic imaging , 2016, UbiComp.

[10]  P. O. Fanger,et al.  Thermal comfort: analysis and applications in environmental engineering, , 1972 .

[11]  Gordon Lightbody,et al.  Prioritised objectives for model predictive control of building heating systems , 2017 .

[12]  Clayton T. Morrison,et al.  Model Predictive Prior Reinforcement Learning for a Heat Pump Thermostat , 2016 .

[13]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[14]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[15]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[16]  Junjing Yang,et al.  Bayesian calibration of building energy models with large datasets , 2017 .

[17]  Srinivasan Keshav,et al.  Optimal Personal Comfort Management Using SPOT+ , 2013, BuildSys@SenSys.

[18]  Zoltán Nagy,et al.  Comprehensive analysis of the relationship between thermal comfort and building control research - A data-driven literature review , 2018 .

[19]  Lei Chen,et al.  A new model predictive control scheme for energy and cost savings in commercial buildings: An airport terminal building case study , 2015 .

[20]  Siliang Lu,et al.  A DEEP REINFORCEMENT LEARNING APPROACH TO USINGWHOLE BUILDING ENERGYMODEL FOR HVAC OPTIMAL CONTROL , 2018 .

[21]  Johan Driesen,et al.  Deep Reinforcement Learning for Optimal Control of Space Heating , 2018, ArXiv.

[22]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis , 2006 .

[23]  Wei Liang,et al.  MPC control for improving energy efficiency of a building air handler for multi-zone VAVs , 2015 .

[24]  K. Dalamagkidisa,et al.  Reinforcement learning for energy conservation and comfort in buildings , 2007 .

[25]  新 雅夫,et al.  ASHRAE(American Society of Heating,Refrigerating and Air-Conditioning Engineers)大会"国際年"行事に参加して , 1975 .

[26]  F. Richard Yu,et al.  A Deep Reinforcement Learning-based Trust Management Scheme for Software-defined Vehicular Networks , 2018, DIVANet'18.

[27]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[28]  Lieve Helsen,et al.  Practical implementation and evaluation of model predictive control for an office building in Brussels , 2016 .

[29]  Vivian Loftness,et al.  Investigation of human body skin temperatures as a bio-signal to indicate overall thermal sensations , 2012 .

[30]  Tianshu Wei,et al.  Deep reinforcement learning for building HVAC control , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[31]  Yonggang Wen,et al.  Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning , 2017, IEEE Transactions on Cybernetics.

[32]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[33]  Lei Yang,et al.  Reinforcement learning for optimal control of low exergy buildings , 2015 .

[34]  Kalyan Veeramachaneni,et al.  Using reinforcement learning to optimize occupant comfort and energy usage in HVAC systems , 2014, J. Ambient Intell. Smart Environ..