A Review of Deep Reinforcement Learning for Smart Building Energy Management

Global buildings account for about 30% of the total energy consumption and carbon emission, raising severe energy and environmental concerns. Therefore, it is significant and urgent to develop novel smart building energy management (SBEM) technologies for the advance of energy efficient and green buildings. However, it is a nontrivial task due to the following challenges. First, it is generally difficult to develop an explicit building thermal dynamics model that is both accurate and efficient enough for building control. Second, there are many uncertain system parameters (e.g., renewable generation output, outdoor temperature, and the number of occupants). Third, there are many spatially and temporally coupled operational constraints. Fourth, building energy optimization problems can not be solved in real time by traditional methods when they have extremely large solution spaces. Fifthly, traditional building energy management methods have respective applicable premises, which means that they have low versatility when confronted with varying building environments. With the rapid development of Internet of Things technology and computation capability, artificial intelligence technology find its significant competence in control and optimization. As a general artificial intelligence technology, deep reinforcement learning (DRL) is promising to address the above challenges. Notably, the recent years have seen the surge of DRL for SBEM. However, there lacks a systematic overview of different DRL methods for SBEM. To fill the gap, this article provides a comprehensive review of DRL for SBEM from the perspective of system scale. In particular, we identify the existing unresolved issues and point out possible future research directions.

[1]  Stephen Xia,et al.  A Deep-Reinforcement-Learning-Based Recommender System for Occupant-Driven Energy Optimization in Commercial Buildings , 2020, IEEE Internet of Things Journal.

[2]  Xue Li,et al.  Stochastic Optimal Energy Management and Pricing for Load Serving Entity With Aggregated TCLs of Smart Buildings: A Stackelberg Game Approach , 2021, IEEE Transactions on Industrial Informatics.

[3]  Jingjing Wang,et al.  Deep-Reinforcement-Learning-Based Autonomous UAV Navigation With Sparse Rewards , 2020, IEEE Internet of Things Journal.

[4]  Ashu Verma,et al.  Time-Coordinated Multienergy Management of Smart Buildings Under Uncertainties , 2019, IEEE Transactions on Industrial Informatics.

[5]  Dusit Niyato,et al.  Demand-Side Scheduling Based on Deep Actor-Critic Learning for Smart Grids , 2020, ArXiv.

[6]  Jiayu Zhou,et al.  Transfer Learning in Deep Reinforcement Learning: A Survey , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  K. Raahemifar,et al.  Assessing the potential of surplus clean power in reducing GHG emissions in the building sector using game theory; a case study of Ontario, Canada , 2019, IET Energy Systems Integration.

[8]  Yan Zheng,et al.  Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Reinforcement Learning Framework , 2019, AAMAS.

[9]  Ali Reza Seifi,et al.  Multiagent Reinforcement Learning for Energy Management in Residential Buildings , 2021, IEEE Transactions on Industrial Informatics.

[10]  Li Xia,et al.  Satisfaction based Q-learning for integrated lighting and blind control , 2016 .

[11]  Tom Schaul,et al.  Rainbow: Combining Improvements in Deep Reinforcement Learning , 2017, AAAI.

[12]  Zicheng Cai,et al.  Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy , 2019, BuildSys@SenSys.

[13]  Kuang-Chin Lu,et al.  Energy optimization associated with thermal comfort and indoor air control via a deep reinforcement learning algorithm , 2019, Building and Environment.

[14]  Zhe Wang,et al.  Reinforcement learning for building controls: The opportunities and challenges , 2020, Applied Energy.

[15]  Viktor K. Prasanna,et al.  A cooperative multi-agent deep reinforcement learning framework for real-time residential load scheduling , 2019, IoTDI.

[16]  Goran Strbac,et al.  Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning , 2020, IEEE Transactions on Smart Grid.

[17]  Yuemin Ding,et al.  Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management , 2020, Applied Energy.

[18]  Junaid Qadir,et al.  Leveraging Machine Learning and Big Data for Smart Buildings: A Comprehensive Survey , 2019, IEEE Access.

[19]  Catherine Rosenberg,et al.  Multiple time-scale model predictive control for thermal comfort in buildings , 2016, e-Energy.

[20]  Demis Hassabis,et al.  Mastering Atari, Go, chess and shogi by planning with a learned model , 2019, Nature.

[21]  Szil'ard Aradi,et al.  Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles , 2020, IEEE Transactions on Intelligent Transportation Systems.

[22]  Xingxing Zhang,et al.  A review of reinforcement learning methodologies for controlling occupant comfort in buildings , 2019, Sustainable Cities and Society.

[23]  Kevin Tomsovic,et al.  Community microgrid scheduling considering building thermal dynamics , 2017, 2017 IEEE Power & Energy Society General Meeting.

[24]  Frank Eliassen,et al.  Distributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids , 2020, IEEE Transactions on Industrial Informatics.

[25]  Zheng O'Neill,et al.  One for Many: Transfer Learning for Building HVAC Control , 2020, BuildSys@SenSys.

[26]  Haibo He,et al.  Residential Energy Management with Deep Reinforcement Learning , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[27]  Tao Jiang,et al.  Deep Reinforcement Learning for Smart Home Energy Management , 2020, IEEE Internet of Things Journal.

[28]  Richard E. Turner,et al.  Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning , 2017, NIPS.

[29]  Wei Feng,et al.  A conditional value-at-risk-based dispatch approach for the energy management of smart buildings with HVAC systems , 2020 .

[30]  Jamil Y. Khan,et al.  Real-Time Load Scheduling, Energy Storage Control and Comfort Management for Grid-Connected Solar Integrated Smart Buildings , 2020 .

[31]  Dae-Hyun Choi,et al.  Energy Management of Smart Home with Home Appliances, Energy Storage System and Electric Vehicle: A Hierarchical Deep Reinforcement Learning Approach , 2020, Sensors.

[32]  Jürgen Schmidhuber,et al.  World Models , 2018, ArXiv.

[33]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[34]  Geert Deconinck,et al.  Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning , 2017, CSEE Journal of Power and Energy Systems.

[35]  Yue Tan,et al.  Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges , 2019, IEEE Communications Surveys & Tutorials.

[36]  Tao Jiang,et al.  Distributed Real-Time HVAC Control for Cost-Efficient Commercial Buildings Under Smart Grid Environment , 2018, IEEE Internet of Things Journal.

[37]  Fei Wang,et al.  Multi-Objective Optimization Model of Source–Load–Storage Synergetic Dispatch for a Building Energy Management System Based on TOU Price Demand Response , 2018, IEEE Transactions on Industry Applications.

[38]  Weizheng Hu.  Transforming thermal comfort model and control in the tropics : a machine-learning approach , 2020 .

[39]  Cathal Hoare,et al.  Environmental and energy performance assessment of buildings using scenario modelling and fuzzy analytic network process , 2019 .

[40]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[41]  Bart De Schutter,et al.  Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[42]  Erik Cambria,et al.  A survey on deep reinforcement learning for audio-based applications , 2021, Artificial Intelligence Review.

[43]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[44]  Marc G. Bellemare,et al.  A Distributional Perspective on Reinforcement Learning , 2017, ICML.

[45]  Damien Ernst,et al.  Deep Reinforcement Learning Solutions for Energy Microgrids Management , 2016 .

[46]  Tomoaki Ohtsuki,et al.  Deep Reinforcement Learning for Economic Dispatch of Virtual Power Plant in Internet of Energy , 2020, IEEE Internet of Things Journal.

[47]  W. Feng,et al.  Scenarios of energy efficiency and CO2 emissions reduction potential in the buildings sector in China to year 2050 , 2018, Nature Energy.

[48]  Geert Deconinck,et al.  Domain Randomization for Demand Response of an Electric Water Heater , 2021, IEEE Transactions on Smart Grid.

[49]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[50]  Renke Huang,et al.  Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning , 2021, ArXiv.

[51]  Hiroaki Nishi,et al.  Airflow Direction Control of Air Conditioners Using Deep Reinforcement Learning , 2020, 2020 SICE International Symposium on Control Systems (SICE ISCS).

[52]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[53]  Yasin Yilmaz,et al.  Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey , 2020, IEEE Transactions on Intelligent Transportation Systems.

[54]  Mohsen Guizani,et al.  Semisupervised Deep Reinforcement Learning in Support of IoT and Smart City Services , 2018, IEEE Internet of Things Journal.

[55]  Edward Corry,et al.  Building performance evaluation using OpenMath and Linked Data , 2018, Energy and Buildings.

[56]  Huaguang Zhang,et al.  Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning , 2019, Energies.

[57]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[58]  Youakim Badr,et al.  Energy-efficient heating control for smart buildings with deep reinforcement learning , 2020 .

[59]  Rui Wang,et al.  Deep Reinforcement Learning for Multiobjective Optimization , 2019, IEEE Transactions on Cybernetics.

[60]  Yuanda Wang,et al.  Deep Reinforcement Learning for Economic Energy Scheduling in Data Center Microgrids , 2019, 2019 IEEE Power & Energy Society General Meeting (PESGM).

[61]  Albert Y. Zomaya,et al.  Reinforcement learning in sustainable energy and electric systems: a survey , 2020, Annu. Rev. Control..

[62]  Tao Jiang,et al.  Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings , 2020, IEEE Transactions on Smart Grid.

[63]  Xin Jin,et al.  Transferable Reinforcement Learning for Smart Homes , 2020 .

[64]  Tianyi Chen,et al.  Realistic Peer-to-Peer Energy Trading Model for Microgrids using Deep Reinforcement Learning , 2019, 2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe).

[65]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[66]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[67]  Tao Chen,et al.  An IoT-Based Thermal Model Learning Framework for Smart Buildings , 2020, IEEE Internet of Things Journal.

[68]  Santiago Grijalva,et al.  A Review of Reinforcement Learning for Autonomous Building Energy Management , 2019, Comput. Electr. Eng..

[69]  Haibo He,et al.  Online Scheduling of a Residential Microgrid via Monte-Carlo Tree Search and a Learned Model , 2020, ArXiv.

[70]  Masayoshi Tomizuka,et al.  Model-free Deep Reinforcement Learning for Urban Autonomous Driving , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[71]  Kazem Sohraby,et al.  IoT Considerations, Requirements, and Architectures for Smart Buildings—Energy Optimization and Next-Generation Building Management Systems , 2017, IEEE Internet of Things Journal.

[72]  Zhiwei Wang,et al.  Community Microgrid Planning Considering Building Thermal Dynamics , 2019, 2019 IEEE Sustainable Power and Energy Conference (iSPEC).

[73]  Khee Poh Lam,et al.  Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning , 2019, Energy and Buildings.

[74]  Wei Xiang,et al.  Dynamic Energy Dispatch Based on Deep Reinforcement Learning in IoT-Driven Smart Isolated Microgrids , 2020, IEEE Internet of Things Journal.

[75]  Ying-Chang Liang,et al.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[76]  Seung Ho Hong,et al.  Demand Response for Home Energy Management Using Reinforcement Learning and Artificial Neural Network , 2019, IEEE Transactions on Smart Grid.

[77]  Andrea Monteriù,et al.  Decision support methodologies and day-ahead optimization for smart building energy management in a dynamic pricing scenario , 2020 .

[78]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[79]  Zhiqiang Wan,et al.  Real-Time Residential Demand Response , 2020, IEEE Transactions on Smart Grid.

[80]  Xiaoqing Han,et al.  Review on the research and practice of deep learning and reinforcement learning in smart grids , 2018, CSEE Journal of Power and Energy Systems.

[81]  Mehdi Bennis,et al.  Optimized Computation Offloading Performance in Virtual Edge Computing Systems Via Deep Reinforcement Learning , 2018, IEEE Internet of Things Journal.

[82]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[83]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[84]  Shane Legg,et al.  Noisy Networks for Exploration , 2017, ICLR.

[85]  Vijay Janapa Reddi,et al.  Deep Reinforcement Learning for Cyber Security , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[86]  Steve Greenberg,et al.  Window operation and impacts on building energy consumption , 2015 .

[87]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[88]  Tianshu Wei,et al.  Deep reinforcement learning for building HVAC control , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[89]  Laurence T. Yang,et al.  Learning-Automata-Based Confident Information Coverage Barriers for Smart Ocean Internet of Things , 2020, IEEE Internet of Things Journal.

[90]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[91]  Victor Talpaert,et al.  Deep Reinforcement Learning for Autonomous Driving: A Survey , 2020, IEEE Transactions on Intelligent Transportation Systems.

[92]  Huan Wang,et al.  Non-invasive (non-contact) measurements of human thermal physiology signals and thermal comfort/discomfort poses -A review , 2020, Energy and Buildings.

[93]  Yuren Zhou,et al.  An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management , 2020 .

[94]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[95]  Robert C. Qiu,et al.  Deep reinforcement learning for power system: An overview , 2019, CSEE Journal of Power and Energy Systems.

[96]  Young Ran Yoon,et al.  Performance based thermal comfort control (PTCC) using deep reinforcement learning for space cooling , 2019, Energy and Buildings.

[97]  Qing-Shan Jia,et al.  Energy-Efficient Buildings Facilitated by Microgrid , 2010, IEEE Transactions on Smart Grid.

[98]  Mary Ann Piette,et al.  Data fusion in predicting internal heat gains for office buildings through a deep learning approach , 2019, Applied Energy.

[99]  Johan Driesen,et al.  Deep Reinforcement Learning for Optimal Control of Space Heating , 2018, ArXiv.

[100]  Takeshi Morinibu,et al.  Application of Deep Reinforcement Learning in Residential Preconditioning for Radiation Temperature , 2019, 2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI).

[101]  Xiaohong Guan,et al.  Optimal Scheduling of Distributed Hydrogen-based Multi-Energy Systems for Building Energy Cost and Carbon Emission Reduction , 2020, 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE).

[102]  Bernardete Ribeiro,et al.  A Survey on Home Energy Management , 2020, IEEE Access.

[103]  Xi Chen,et al.  Meta-Learning for Multi-objective Reinforcement Learning , 2018, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[104]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[105]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[106]  Saeid Nahavandi,et al.  Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications , 2018, ArXiv.

[107]  David E. Culler,et al.  Energy-Efficient Building HVAC Control Using Hybrid System LBMPC , 2012, ArXiv.

[108]  C. Rasmussen,et al.  Improving PILCO with Bayesian Neural Network Dynamics Models , 2016 .

[109]  Wayes Tushar,et al.  A Survey of Computational Intelligence Techniques for Air-Conditioners Energy Management , 2020, IEEE Transactions on Emerging Topics in Computational Intelligence.

[110]  H. Gooi,et al.  Optimization strategy based on deep reinforcement learning for home energy management , 2020 .

[111]  Francesco Borrelli,et al.  Stochastic Model Predictive Control for Building HVAC Systems: Complexity and Conservatism , 2015, IEEE Transactions on Control Systems Technology.

[112]  Semiha Ergan,et al.  Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network , 2020 .

[113]  Javier García-González,et al.  Optimising a Microgrid System by Deep Reinforcement Learning Techniques , 2020, Energies.

[114]  Ivana Dusparic,et al.  Multi-agent Deep Reinforcement Learning for Zero Energy Communities , 2018, 2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe).

[115]  Xiangyu Zhang,et al.  An Edge-Cloud Integrated Solution for Buildings Demand Response Using Reinforcement Learning , 2021, IEEE Transactions on Smart Grid.

[116]  Yu Wang,et al.  The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games , 2021, NeurIPS.

[117]  Alberto Cerpa,et al.  OCTOPUS: Deep Reinforcement Learning for Holistic Smart Building Control , 2019, BuildSys.

[118]  Dongbin Zhao,et al.  A Survey of Deep Reinforcement Learning in Video Games , 2019, ArXiv.

[119]  Anand Sivasubramaniam,et al.  MARCO - Multi-Agent Reinforcement learning based COntrol of building HVAC systems , 2020, e-Energy.

[120]  Sean P. Meyn,et al.  Reinforcement Learning for Control of Building HVAC Systems , 2020, 2020 American Control Conference (ACC).

[121]  Mugen Peng,et al.  Deep Reinforcement Learning-Based Mode Selection and Resource Management for Green Fog Radio Access Networks , 2018, IEEE Internet of Things Journal.

[122]  Joshua B. Tenenbaum,et al.  Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation , 2016, NIPS.

[123]  Zhanhong Jiang,et al.  Deep Transfer Learning for Thermal Dynamics Modeling in Smart Buildings , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[124]  Hyuk Lim,et al.  Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings , 2018, Energies.