Deep Reinforcement Learning for Smart Building Energy Management: A Survey.

Global buildings consumed 30\% of total energy and generated 28\% of total carbon emission in 2018, which leads to economic and environmental concerns. Therefore, it is of great significance to reduce energy consumption, energy cost and carbon emission of buildings while maintaining user comfort. To this end, several challenges have to be addressed. Firstly, it is very challenging to develop a building thermal dynamics model that is both accurate and efficient enough for building control. Secondly, there are many kinds of uncertainties. Thirdly, there are many spatially and temporally operational constraints. Fourthly, building energy optimization problems may have extremely large solution spaces, which can not be solved in real-time by traditional methods. Fifthly, traditional building energy management methods have respective applicable premises, which means that they have low versatility when confronted with varying building environments. Since deep reinforcement learning (DRL) is a promising tool to address the above challenges, this paper presents a comprehensive literature review on DRL for smart building energy management (SBEM). To be specific, we first introduce the fundamentals of DRL and provide the classification of DRL methods used in existing works related to SBEM. Then, we review the applications of DRL in a single building energy subsystem, multiple energy subsystems of buildings, and building microgrids, respectively. Furthermore, we identify the unsolved issues and point out the possible research directions of applying DRL. Finally, we summarize the lessons learned from this survey.

[1]  Goran Strbac,et al.  Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning , 2020, IEEE Transactions on Smart Grid.

[2]  Mary Ann Piette,et al.  Data fusion in predicting internal heat gains for office buildings through a deep learning approach , 2019, Applied Energy.

[3]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[4]  Catherine Rosenberg,et al.  Multiple time-scale model predictive control for thermal comfort in buildings , 2016, e-Energy.

[5]  Stephen Xia,et al.  A Deep-Reinforcement-Learning-Based Recommender System for Occupant-Driven Energy Optimization in Commercial Buildings , 2020, IEEE Internet of Things Journal.

[6]  David E. Culler,et al.  Energy-Efficient Building HVAC Control Using Hybrid System LBMPC , 2012, ArXiv.

[7]  Ivana Dusparic,et al.  Multi-agent Deep Reinforcement Learning for Zero Energy Communities , 2018, 2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe).

[8]  Hiroaki Nishi,et al.  Airflow Direction Control of Air Conditioners Using Deep Reinforcement Learning , 2020, 2020 SICE International Symposium on Control Systems (SICE ISCS).

[9]  Zicheng Cai,et al.  Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy , 2019, BuildSys@SenSys.

[10]  Fei Sha,et al.  Actor-Attention-Critic for Multi-Agent Reinforcement Learning , 2018, ICML.

[11]  Xiangyu Zhang,et al.  An Edge-Cloud Integrated Solution for Buildings Demand Response Using Reinforcement Learning , 2021, IEEE Transactions on Smart Grid.

[12]  Tao Jiang,et al.  Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings , 2021, IEEE Transactions on Smart Grid.

[13]  Xiaoqing Han,et al.  Review on the research and practice of deep learning and reinforcement learning in smart grids , 2018, CSEE Journal of Power and Energy Systems.

[14]  Tom Schaul,et al.  Prioritized Experience Replay , 2015, ICLR.

[15]  Yuemin Ding,et al.  Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management , 2020, Applied Energy.

[16]  Dusit Niyato,et al.  Demand-Side Scheduling Based on Deep Actor-Critic Learning for Smart Grids , 2020, ArXiv.

[17]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[18]  Haibo He,et al.  Online Scheduling of a Residential Microgrid via Monte-Carlo Tree Search and a Learned Model , 2020, ArXiv.

[19]  Edward Corry,et al.  Building performance evaluation using OpenMath and Linked Data , 2018, Energy and Buildings.

[20]  Steve Greenberg,et al.  Window operation and impacts on building energy consumption , 2015 .

[21]  Sean P. Meyn,et al.  Reinforcement Learning for Control of Building HVAC Systems , 2020, 2020 American Control Conference (ACC).

[22]  Jamil Y. Khan,et al.  Real-Time Load Scheduling, Energy Storage Control and Comfort Management for Grid-Connected Solar Integrated Smart Buildings , 2020 .

[23]  Xingxing Zhang,et al.  A review of reinforcement learning methodologies for controlling occupant comfort in buildings , 2019, Sustainable Cities and Society.

[24]  Weizheng Hu.  Transforming thermal comfort model and control in the tropics : a machine-learning approach , 2020 .

[25]  Xiaohong Guan,et al.  Optimal Scheduling of Distributed Hydrogen-based Multi-Energy Systems for Building Energy Cost and Carbon Emission Reduction , 2020, 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE).

[26]  Cathal Hoare,et al.  Environmental and energy performance assessment of buildings using scenario modelling and fuzzy analytic network process , 2019 .

[27]  Semiha Ergan,et al.  Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network , 2020 .

[28]  W. Feng,et al.  Scenarios of energy efficiency and CO2 emissions reduction potential in the buildings sector in China to year 2050 , 2018, Nature Energy.

[29]  Tianshu Wei,et al.  Deep reinforcement learning for building HVAC control , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[30]  Zhanhong Jiang,et al.  Deep Transfer Learning for Thermal Dynamics Modeling in Smart Buildings , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[31]  Ashu Verma,et al.  Time-Coordinated Multienergy Management of Smart Buildings Under Uncertainties , 2019, IEEE Transactions on Industrial Informatics.

[32]  Khee Poh Lam,et al.  Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning , 2019, Energy and Buildings.

[33]  Ali Reza Seifi,et al.  Multiagent Reinforcement Learning for Energy Management in Residential Buildings , 2021, IEEE Transactions on Industrial Informatics.

[34]  Demis Hassabis,et al.  Mastering Atari, Go, chess and shogi by planning with a learned model , 2019, Nature.

[35]  Canbing Li,et al.  An Optimized EV Charging Model Considering TOU Price and SOC Curve , 2012, IEEE Transactions on Smart Grid.

[36]  Frank Eliassen,et al.  Distributed Deep Reinforcement Learning for Intelligent Load Scheduling in Residential Smart Grids , 2020, IEEE Transactions on Industrial Informatics.

[37]  Huan Wang,et al.  Non-invasive (non-contact) measurements of human thermal physiology signals and thermal comfort/discomfort poses -A review , 2020, Energy and Buildings.

[38]  Geert Deconinck,et al.  Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning , 2017, CSEE Journal of Power and Energy Systems.

[39]  Bart De Schutter,et al.  Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[40]  Youakim Badr,et al.  Energy-efficient heating control for smart buildings with deep reinforcement learning , 2020 .

[41]  Xi Chen,et al.  Meta-Learning for Multi-objective Reinforcement Learning , 2018, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  Bernardete Ribeiro,et al.  A Survey on Home Energy Management , 2020, IEEE Access.

[43]  Xue Li,et al.  Stochastic Optimal Energy Management and Pricing for Load Serving Entity With Aggregated TCLs of Smart Buildings: A Stackelberg Game Approach , 2021, IEEE Transactions on Industrial Informatics.

[44]  Tao Chen,et al.  An IoT-Based Thermal Model Learning Framework for Smart Buildings , 2020, IEEE Internet of Things Journal.

[45]  Francesco Borrelli,et al.  Stochastic Model Predictive Control for Building HVAC Systems: Complexity and Conservatism , 2015, IEEE Transactions on Control Systems Technology.

[46]  Zhiwei Wang,et al.  Community Microgrid Planning Considering Building Thermal Dynamics , 2019, 2019 IEEE Sustainable Power and Energy Conference (iSPEC).

[47]  Yan Zheng,et al.  Large-Scale Home Energy Management Using Entropy-Based Collective Multiagent Reinforcement Learning Framework , 2019, AAMAS.

[48]  Yuanda Wang,et al.  Deep Reinforcement Learning for Economic Energy Scheduling in Data Center Microgrids , 2019, 2019 IEEE Power & Energy Society General Meeting (PESGM).

[49]  Seung Ho Hong,et al.  Demand Response for Home Energy Management Using Reinforcement Learning and Artificial Neural Network , 2019, IEEE Transactions on Smart Grid.

[50]  Wei Feng,et al.  A conditional value-at-risk-based dispatch approach for the energy management of smart buildings with HVAC systems , 2020 .

[51]  Qing-Shan Jia,et al.  Energy-Efficient Buildings Facilitated by Microgrid , 2010, IEEE Transactions on Smart Grid.

[52]  Rose Qingyang Hu,et al.  Mobility-Aware Edge Caching and Computing in Vehicle Networks: A Deep Reinforcement Learning , 2018, IEEE Transactions on Vehicular Technology.

[53]  Junaid Qadir,et al.  Leveraging Machine Learning and Big Data for Smart Buildings: A Comprehensive Survey , 2019, IEEE Access.

[54]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[55]  Robert C. Qiu,et al.  Deep reinforcement learning for power system: An overview , 2019, CSEE Journal of Power and Energy Systems.

[56]  Saeid Nahavandi,et al.  Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications , 2018, IEEE Transactions on Cybernetics.

[57]  Zhiqiang Wan,et al.  Real-Time Residential Demand Response , 2020, IEEE Transactions on Smart Grid.

[58]  Kazem Sohraby,et al.  IoT Considerations, Requirements, and Architectures for Smart Buildings—Energy Optimization and Next-Generation Building Management Systems , 2017, IEEE Internet of Things Journal.

[59]  Young Ran Yoon,et al.  Performance based thermal comfort control (PTCC) using deep reinforcement learning for space cooling , 2019, Energy and Buildings.

[60]  Kuang-Chin Lu,et al.  Energy optimization associated with thermal comfort and indoor air control via a deep reinforcement learning algorithm , 2019, Building and Environment.

[61]  Wayes Tushar,et al.  A Survey of Computational Intelligence Techniques for Air-Conditioners Energy Management , 2020, IEEE Transactions on Emerging Topics in Computational Intelligence.

[62]  Rui Wang,et al.  Deep Reinforcement Learning for Multiobjective Optimization , 2019, IEEE Transactions on Cybernetics.

[63]  Shing-Chow Chan,et al.  Demand Response Optimization for Smart Home Scheduling Under Real-Time Pricing , 2012, IEEE Transactions on Smart Grid.

[64]  Damien Ernst,et al.  Deep Reinforcement Learning Solutions for Energy Microgrids Management , 2016 .

[65]  Javier García-González,et al.  Optimising a Microgrid System by Deep Reinforcement Learning Techniques , 2020, Energies.

[66]  Santiago Grijalva,et al.  A Review of Reinforcement Learning for Autonomous Building Energy Management , 2019, Comput. Electr. Eng..

[67]  Viktor K. Prasanna,et al.  A cooperative multi-agent deep reinforcement learning framework for real-time residential load scheduling , 2019, IoTDI.

[68]  Joshua B. Tenenbaum,et al.  Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation , 2016, NIPS.

[69]  Takeshi Morinibu,et al.  Application of Deep Reinforcement Learning in Residential Preconditioning for Radiation Temperature , 2019, 2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI).

[70]  Masayoshi Tomizuka,et al.  Model-free Deep Reinforcement Learning for Urban Autonomous Driving , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[71]  Antonio Liotta,et al.  On-Line Building Energy Optimization Using Deep Reinforcement Learning , 2017, IEEE Transactions on Smart Grid.

[72]  Li Xia,et al.  Satisfaction based Q-learning for integrated lighting and blind control , 2016 .

[73]  Alberto Cerpa,et al.  OCTOPUS: Deep Reinforcement Learning for Holistic Smart Building Control , 2019, BuildSys.

[74]  Wei Xiang,et al.  Dynamic Energy Dispatch in Isolated Microgrids Based on Deep Reinforcement Learning , 2020, ArXiv.

[75]  Vijay Janapa Reddi,et al.  Deep Reinforcement Learning for Cyber Security , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[76]  Andrea Monteriù,et al.  Decision support methodologies and day-ahead optimization for smart building energy management in a dynamic pricing scenario , 2020 .

[77]  Thanh Thi Nguyen,et al.  A Multi-Objective Deep Reinforcement Learning Framework , 2018, Eng. Appl. Artif. Intell..

[78]  Dae-Hyun Choi,et al.  Energy Management of Smart Home with Home Appliances, Energy Storage System and Electric Vehicle: A Hierarchical Deep Reinforcement Learning Approach , 2020, Sensors.

[79]  Zheng O'Neill,et al.  One for Many: Transfer Learning for Building HVAC Control , 2020, BuildSys@SenSys.

[80]  Haibo He,et al.  Residential Energy Management with Deep Reinforcement Learning , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[81]  Ying-Chang Liang,et al.  Applications of Deep Reinforcement Learning in Communications and Networking: A Survey , 2018, IEEE Communications Surveys & Tutorials.

[82]  Albert Y. Zomaya,et al.  Reinforcement learning in sustainable energy and electric systems: a survey , 2020, Annu. Rev. Control..

[83]  Huaguang Zhang,et al.  Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning , 2019, Energies.

[84]  Zhe Wang,et al.  Reinforcement learning for building controls: The opportunities and challenges , 2020, Applied Energy.

[85]  Fei Wang,et al.  Multi-Objective Optimization Model of Source–Load–Storage Synergetic Dispatch for a Building Energy Management System Based on TOU Price Demand Response , 2018, IEEE Transactions on Industry Applications.

[86]  Kevin Tomsovic,et al.  Community microgrid scheduling considering building thermal dynamics , 2017, 2017 IEEE Power & Energy Society General Meeting.

[87]  Yuren Zhou,et al.  An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management , 2020 .

[88]  Tianyi Chen,et al.  Realistic Peer-to-Peer Energy Trading Model for Microgrids using Deep Reinforcement Learning , 2019, 2019 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe).

[89]  Yue Tan,et al.  Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges , 2019, IEEE Communications Surveys & Tutorials.

[90]  Catherine Rosenberg,et al.  On the interaction between personal comfort systems and centralized HVAC systems in office buildings , 2017, Advances in Building Energy Research.

[91]  Johan Driesen,et al.  Deep Reinforcement Learning for Optimal Control of Space Heating , 2018, ArXiv.

[92]  Hyuk Lim,et al.  Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings , 2018, Energies.

[93]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[94]  José R. Vázquez-Canteli,et al.  Reinforcement learning for demand response: A review of algorithms and modeling techniques , 2019, Applied Energy.

[95]  Dongbin Zhao,et al.  A Survey of Deep Reinforcement Learning in Video Games , 2019, ArXiv.

[96]  Vincent W. S. Wong,et al.  Autonomous Demand-Side Management Based on Game-Theoretic Energy Consumption Scheduling for the Future Smart Grid , 2010, IEEE Transactions on Smart Grid.

[97]  Tao Jiang,et al.  Deep Reinforcement Learning for Smart Home Energy Management , 2020, IEEE Internet of Things Journal.

[98]  Richard E. Turner,et al.  Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning , 2017, NIPS.

[99]  Tao Jiang,et al.  Distributed Real-Time HVAC Control for Cost-Efficient Commercial Buildings Under Smart Grid Environment , 2018, IEEE Internet of Things Journal.

[100]  Anand Sivasubramaniam,et al.  MARCO - Multi-Agent Reinforcement learning based COntrol of building HVAC systems , 2020, e-Energy.

[101]  Laurence T. Yang,et al.  Learning-Automata-Based Confident Information Coverage Barriers for Smart Ocean Internet of Things , 2020, IEEE Internet of Things Journal.