Collaborative energy demand response with decentralized actor and centralized critic

The ongoing industrialization and rising technology adoption around the world are leading to ever higher energy consumption. The benefits of electrification are enormous, but the growing demand also comes with challenges with respect to associated greenhouse gas emissions. Although continuing progress in energy research has brought up new technologies in energy generation, storage, and distribution, most of those technologies focus on increasing efficiency of individual components. Work on integration and coordination abilities between individual components in micro-grids will lead to further improvements and gains in efficiency that are necessary to reduce carbon footprints and slow down climate change. To this end, the CityLearn environment provides a simulation framework that allows the control of energy components in buildings that are organized in districts. In this paper, we propose an energy management system based on the decentralized actor-critic reinforcement learning algorithm MARLISA but integrate a centralized critic and call it MARLISADSCC. In this way, we are training a model to autonomously control the energy storage of individual buildings in a CityLearn district to improve demand response guided by a better informed training signal. We show performance increases over baseline control techniques for a district but also discuss the resulting action selection for individual buildings.

[1]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[2]  Henrik Sandberg,et al.  A Survey of Distributed Optimization and Control Algorithms for Electric Power Systems , 2017, IEEE Transactions on Smart Grid.

[3]  G. Goldman,et al.  A Survey of Utility Experience with Real Time Pricing , 2004 .

[4]  Bart De Schutter,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[5]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[6]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.

[7]  Reinaldo A. C. Bianchi,et al.  DECAF: Deep Case-based Policy Inference for knowledge transfer in Reinforcement Learning , 2020, Expert Syst. Appl..

[8]  Steven G. Smith,et al.  A federated simulation toolkit for electric power grid and communication network co-simulation , 2015, 2015 Workshop on Modeling and Simulation of Cyber-Physical Energy Systems (MSCPES).

[9]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[10]  Ruben Glatt,et al.  Increasing performance of electric vehicles in ride-hailing services using deep reinforcement learning , 2019, ArXiv.

[11]  Tamer Basar,et al.  Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms , 2019, Handbook of Reinforcement Learning and Control.

[12]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[13]  Lakhmi C. Jain,et al.  Innovations in Multi-Agent Systems and Applications - 1 , 2010 .

[14]  Daniel Pérez Palomar,et al.  Demand-Side Management via Distributed Energy Generation and Storage Optimization , 2013, IEEE Transactions on Smart Grid.

[15]  José R. Vázquez-Canteli,et al.  Coordinated energy management for a cluster of buildings through deep reinforcement learning , 2021 .

[16]  Andrey Bernstein,et al.  Real-Time Feedback-Based Optimization of Distribution Grids: A Unified Approach , 2017, IEEE Transactions on Control of Network Systems.

[17]  Makbul A.M. Ramli,et al.  A review of optimization approaches for hybrid distributed energy generation systems: Off-grid and grid-connected systems , 2018 .

[18]  José R. Vázquez-Canteli,et al.  CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management , 2020, ArXiv.

[19]  Anna Helena Reali Costa,et al.  Intelligent Decision-Making for Smart Home Energy Management , 2014, Journal of Intelligent & Robotic Systems.

[20]  Alberto Bemporad,et al.  Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities , 2018 .

[21]  José R. Vázquez-Canteli,et al.  The CityLearn Challenge 2020 , 2020, BuildSys@SenSys.

[22]  Zhe Wang,et al.  Reinforcement learning for building controls: The opportunities and challenges , 2020, Applied Energy.

[23]  Felipe Leno da Silva,et al.  Coordination of Electric Vehicle Charging Through Multiagent Reinforcement Learning , 2020, IEEE Transactions on Smart Grid.

[24]  Santiago Grijalva,et al.  A Review of Reinforcement Learning for Autonomous Building Energy Management , 2019, Comput. Electr. Eng..

[25]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[26]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Goran Strbac,et al.  Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach , 2021, Applied Energy.

[28]  José R. Vázquez-Canteli,et al.  MARLISA: Multi-Agent Reinforcement Learning with Iterative Sequential Action Selection for Load Shaping of Grid-Interactive Connected Buildings , 2020, BuildSys@SenSys.