Optimal control of HVAC and window systems for natural ventilation through reinforcement learning

Abstract Natural ventilation is a green building strategy that improves building energy efficiency, indoor thermal environment, and air quality. However, in practice, it is not always clear when and how to utilize the natural ventilation and coordinate its operation with the HVAC system. This paper introduces a reinforcement learning control strategy, specifically through model-free Q-learning, that makes optimal control decisions for HVAC and window systems to minimize both energy consumption and thermal discomfort. This control system evaluates the outdoor and indoor environments (temperature, humidity, solar radiation, and wind speed) at each time step, and responds with the best control decision that targets both immediate and long-term goals. The reinforcement learning control is evaluated through numerical simulation on a building thermal model and compared with a rule-based heuristic control strategy. Case studies in hot-and-humid Miami and warm-and-mild Los Angeles demonstrated the superior performance of reinforcement learning control, which led to 13% and 23% lower HVAC system energy consumption, 62% and 80% lower discomfort degree hours, and 63% and 77% fewer high humidity hours compared to heuristic control. Unlike heuristic control that requires specific knowledge of individual buildings and the creation of exhaustive decision-making scenarios to improve performance, reinforcement learning control guarantees optimality through self-advancement on given goals and cost functions and is able to adapt to stochastic occupancy and occupant behaviors, which is difficult to accommodate by heuristic control.

[1]  Martin Tenpierik,et al.  Review of the impact of urban block form on thermal performance, solar access and ventilation , 2014 .

[2]  Holly Wasilowski Samuelson,et al.  The impact of window opening and other occupant behavior on simulated energy performance in residence halls , 2017 .

[3]  Vlad Isakov,et al.  Roadside vegetation barrier designs to mitigate near-road air pollution impacts. , 2016, The Science of the total environment.

[4]  C.-F. Gao,et al.  Evaluating the influence of openings configuration on natural ventilation performance of residential , 2011 .

[5]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[6]  Mahroo Eftekhari,et al.  Application of fuzzy control in naturally ventilated buildings for summer conditions , 2003 .

[7]  Ali Malkawi,et al.  Estimating natural ventilation potential for high-rise buildings considering boundary layer meteorology , 2017 .

[8]  Bert Blocken,et al.  Coupled urban wind flow and indoor natural ventilation modelling on a high-resolution grid: A case study for the Amsterdam ArenA stadium , 2010, Environ. Model. Softw..

[9]  H. Shetabivash,et al.  Investigation of opening position and shape on the natural cross ventilation , 2015 .

[10]  Andrew J Landers,et al.  Quantifying the effect of vegetation on near-road air quality using brief campaigns. , 2015, Environmental pollution.

[11]  Khairul Salleh Mohamed Sahari,et al.  Energy saving by integrated control of natural ventilation and HVAC systems using model guide for comparison , 2014 .

[12]  Zheming Tong,et al.  Microenvironmental air quality impact of a commercial-scale biomass heating system. , 2017, Environmental pollution.

[13]  Leon R. Glicksman,et al.  Design analysis of single-sided natural ventilation , 2003 .

[14]  Arnold Janssens,et al.  Passive cooling in a low-energy office building , 2005 .

[15]  Shui Yuan,et al.  Multiple-zone ventilation and temperature control of a single-duct VAV system using model predictive strategy , 2006 .

[16]  Xiwang Li,et al.  Building energy consumption on-line forecasting using physics based system identification , 2014 .

[17]  Nathan Mendes,et al.  Predictive controllers for thermal comfort optimization and energy savings , 2008 .

[18]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 1. Theoretical foundation , 2006 .

[19]  C. J. Koinakis,et al.  Combined thermal and natural ventilation modeling for long-term energy assessment: validation with experimental measurements , 2005 .

[20]  Jin Wen,et al.  Review of building energy modeling for control and operation , 2014 .

[21]  Lei Yang,et al.  Reinforcement learning for optimal control of low exergy buildings , 2015 .

[22]  P. A. Østergaard,et al.  Energy saving potential of utilizing natural ventilation under warm conditions – A case study of Mexico , 2014 .

[23]  Qingyan Chen,et al.  Natural Ventilation Design for Houses in Thailand , 2001 .

[24]  Teresa Wu,et al.  Short-term building energy model recommendation system: A meta-learning approach , 2016 .

[25]  Kaamran Raahemifar,et al.  Artificial neural network (ANN) based model predictive control (MPC) and optimization of HVAC systems: A state of the art review and case study of a residential HVAC system , 2017 .

[26]  Jiun-Jih Miau,et al.  Wind driven natural ventilation through multiple windows of a building: A computational approach , 2012 .

[27]  D. Kolokotsa,et al.  Predictive control techniques for energy and indoor environmental quality management in buildings , 2009 .

[28]  Yujiao Chen,et al.  Integrated design workflow and a new tool for urban rainwater management. , 2015, Journal of environmental management.

[29]  Till Pasquay,et al.  Natural ventilation in high-rise buildings with double facades, saving or waste of energy , 2004 .

[30]  Gail Brager,et al.  Mixed-mode cooling. , 2006 .

[31]  Can Cui,et al.  A recommendation system for meta-modeling: A meta-learning based approach , 2016, Expert Syst. Appl..

[32]  Simeng Liu,et al.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis , 2006 .

[33]  D. Kolokotsa,et al.  Reinforcement learning for energy conservation and comfort in buildings , 2007 .

[34]  Yungang Wang,et al.  Modeling multi-scale aerosol dynamics and micro-environmental air quality near a large highway intersection using the CTAG model. , 2013, The Science of the total environment.

[35]  Neil Hirst,et al.  Buildings and Climate Change , 2013 .

[36]  Li Xia,et al.  Satisfaction based Q-learning for integrated lighting and blind control , 2016 .

[37]  Ali Malkawi,et al.  Defining the Influence Region in neighborhood-scale CFD simulations for natural ventilation design , 2016 .

[38]  Bin Yan,et al.  Predicting thermal and energy performance of mixed-mode ventilation using an integrated simulation approach , 2016 .

[39]  Er-Wei Bai,et al.  Developing a whole building cooling energy forecasting model for on-line operation optimization using proactive system identification , 2016 .

[40]  Holly Wasilowski Samuelson,et al.  Parametric Energy Simulation in Early Design: High-Rise Residential Buildings in Urban Contexts , 2016 .

[41]  Ali Malkawi,et al.  Investigating natural ventilation potentials across the globe: Regional and climatic variations , 2017 .

[42]  Zhe-ming Tong,et al.  A case study of air quality above an urban roof top vegetable farm. , 2016, Environmental pollution.

[43]  Xiwang Li,et al.  System identification and data fusion for on-line adaptive energy forecasting in virtual and real commercial buildings , 2016 .