On-Line Building Energy Optimization Using Deep Reinforcement Learning

Unprecedented high volumes of data are becoming available with the growth of the advanced metering infrastructure. These are expected to benefit planning and operation of the future power systems and to help customers transition from a passive to an active role. In this paper, we explore for the first time in the smart grid context the benefits of using deep reinforcement learning, a hybrid type of methods that combines reinforcement learning with deep learning, to perform on-line optimization of schedules for building energy management systems. The learning procedure was explored using two methods, Deep Q-learning and deep policy gradient, both of which have been extended to perform multiple actions simultaneously. The proposed approach was validated on the large-scale Pecan Street Inc. database. This highly dimensional database includes information about photovoltaic power generation, electric vehicles and buildings appliances. Moreover, these on-line energy scheduling strategies could be used to provide real-time feedback to consumers to encourage more efficient use of electricity.

[1]  Damien Ernst,et al.  Deep Reinforcement Learning Solutions for Energy Microgrids Management , 2016 .

[2]  Milos Manic,et al.  Intelligent Buildings of the Future: Cyberaware, Deep Learning Powered, and Human Interacting , 2016, IEEE Industrial Electronics Magazine.

[3]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[4]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[5]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[7]  Wil L. Kling,et al.  Comparison of machine learning methods for estimating energy consumption in buildings , 2014, 2014 International Conference on Probabilistic Methods Applied to Power Systems (PMAPS).

[8]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[9]  Antonio Capone,et al.  Optimization Models and Methods for Demand-Side Management of Residential Users: A Survey , 2014 .

[10]  João P. S. Catalão,et al.  Coordinated Operation of a Neighborhood of Smart Households Comprising Electric Vehicles, Energy Storage and Distributed Generation , 2016, IEEE Transactions on Smart Grid.

[11]  Hamed Mohsenian Rad,et al.  Optimal Residential Load Control With Price Prediction in Real-Time Electricity Pricing Environments , 2010, IEEE Transactions on Smart Grid.

[12]  Yitao Liu,et al.  Deep learning based ensemble approach for probabilistic wind power forecasting , 2017 .

[13]  Marco Levorato,et al.  Residential Demand Response Using Reinforcement Learning , 2010, 2010 First IEEE International Conference on Smart Grid Communications.

[14]  Daniel L. Marino,et al.  Building energy load forecasting using Deep Neural Networks , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[15]  Stephen P. Boyd,et al.  Dynamic Network Energy Management via Proximal Message Passing , 2013, Found. Trends Optim..

[16]  Mihaela van der Schaar,et al.  Dynamic Pricing and Energy Consumption Scheduling With Reinforcement Learning , 2016, IEEE Transactions on Smart Grid.

[17]  Stefan Schaal,et al.  2008 Special Issue: Reinforcement learning of motor skills with policy gradients , 2008 .

[18]  Hongseok Kim,et al.  Deep neural network based demand side short term load forecasting , 2016, 2016 IEEE International Conference on Smart Grid Communications (SmartGridComm).

[19]  Louis Wehenkel,et al.  Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Jin Wei,et al.  Real-Time Detection of False Data Injection Attacks in Smart Grid: A Deep Learning-Based Intelligent Mechanism , 2017, IEEE Transactions on Smart Grid.

[21]  Eduardo F. Morales,et al.  An Introduction to Reinforcement Learning , 2011 .

[22]  Wil L. Kling,et al.  Comfort-constrained demand flexibility management for building aggregations using a decentralized approach , 2015, 2015 International Conference on Smart Cities and Green ICT Systems (SMARTGREENS).

[23]  Madeleine Gibescu,et al.  Deep learning for estimating building energy consumption , 2016 .

[24]  Sergey Levine,et al.  High-Dimensional Continuous Control Using Generalized Advantage Estimation , 2015, ICLR.

[25]  Ronnie Belmans,et al.  Demand response of a heterogeneous cluster of electric water heaters using batch reinforcement learning , 2014, 2014 Power Systems Computation Conference.

[26]  Madeleine Gibescu,et al.  Unsupervised energy prediction in a Smart Grid context using reinforcement cross-building transfer learning , 2016 .

[27]  Ran Li,et al.  Deep Learning for Household Load Forecasting—A Novel Pooling Deep RNN , 2018, IEEE Transactions on Smart Grid.

[28]  Anna Helena Reali Costa,et al.  Batch Reinforcement Learning for Smart Home Energy Management , 2015, IJCAI.

[29]  Geoffrey E. Hinton,et al.  On rectified linear units for speech processing , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[30]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[31]  Thomas Kunz,et al.  Computational Methods for Residential Energy Cost Optimization in Smart Grids , 2016, ACM Comput. Surv..

[32]  Christos V. Verikoukis,et al.  A Survey on Demand Response Programs in Smart Grids: Pricing Methods and Optimization Algorithms , 2015, IEEE Communications Surveys & Tutorials.

[33]  Eric Eaton,et al.  Online Multi-Task Learning for Policy Gradient Methods , 2014, ICML.

[34]  Josh Bode,et al.  Measuring Short-term Air Conditioner Demand Reductions for Operations and Settlement , 2013 .

[35]  Madeleine Gibescu,et al.  Demand forecasting at low aggregation levels using Factored Conditional Restricted Boltzmann Machine , 2016, 2016 Power Systems Computation Conference (PSCC).

[36]  C. Dent,et al.  Decentralized Multi-Period Economic Dispatch for Real-Time Flexible Demand Management , 2016, IEEE Transactions on Power Systems.

[37]  Nikolaos G. Paterakis,et al.  Deep learning versus traditional machine learning methods for aggregated energy demand prediction , 2017, 2017 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe).