Data-Driven Learning and Load Ensemble Control

Demand response (DR) programs aim to engage distributed small-scale flexible loads, such as thermostatically controllable loads (TCLs), to provide various grid support services. Linearly Solvable Markov Decision Process (LS-MDP), a variant of the traditional MDP, is used to model aggregated TCLs. Then, a model-free reinforcement learning technique called Z-learning is applied to learn the value function and derive the optimal policy for the DR aggregator to control TCLs. The learning process is robust against uncertainty that arises from estimating the passive dynamics of the aggregated TCLs. The efficiency of this data-driven learning is demonstrated through simulations on Heating, Cooling & Ventilation (HVAC) units in a testbed neighborhood of residential houses.

[1]  Tariq Samad,et al.  Automated Demand Response for Smart Buildings and Microgrids: The State of the Practice and Research Challenges , 2016, Proceedings of the IEEE.

[2]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[3]  Paul Raftery,et al.  A review of methods to match building energy simulation models to measured data , 2014 .

[4]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[5]  Michael Chertkov,et al.  Optimal Ensemble Control of Loads in Distribution Grids with Network Constraints , 2017, 2018 Power Systems Computation Conference (PSCC).

[6]  Johanna L. Mathieu,et al.  Uncertainty in the flexibility of aggregations of demand response resources , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.

[7]  Ana Busic,et al.  Ordinary Differential Equation Methods for Markov Decision Processes and Application to Kullback-Leibler Control Cost , 2018, SIAM J. Control. Optim..

[8]  Ning Lu,et al.  A state-queueing model of thermostatically controlled appliances , 2004, IEEE Transactions on Power Systems.

[9]  Duncan S. Callaway.  Tapping the energy storage potential in electric loads to deliver load following and regulation, with application to wind energy , 2009 .

[10]  E. Todorov,et al.  Linearly Solvable Optimal Control , 2013 .

[11]  James P. Bagrow,et al.  Crowdsourcing Predictors of Residential Electric Energy Usage , 2017, IEEE Systems Journal.

[12]  Bowen Li,et al.  Chance Constrained Reserve Scheduling Using Uncertain Controllable Loads Part I: Formulation and Scenario-Based Analysis , 2019, IEEE Transactions on Smart Grid.

[13]  Michael Chertkov,et al.  Optimal Load Ensemble Control in Chance-Constrained Optimal Power Flow , 2019, IEEE Transactions on Smart Grid.

[14]  T. Henzinger,et al.  Model-Checking ω-Regular Properties of Interval Markov Chains , 2008 .

[15]  Michael I. Jordan,et al.  MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY and CENTER FOR BIOLOGICAL AND COMPUTATIONAL LEARNING DEPARTMENT OF BRAIN AND COGNITIVE SCIENCES , 1996 .

[16]  R. Bellman.  A Markovian Decision Process , 1957 .

[17]  Vicenç Gómez,et al.  Fast rates for online learning in Linearly Solvable Markov Decision Processes , 2017, COLT.

[18]  Johanna L. Mathieu,et al.  Modeling and Control of Aggregated Heterogeneous Thermostatically Controlled Loads for Ancillary Services , 2011 .

[19]  Ian A. Hiskens,et al.  Achieving Controllability of Electric Loads , 2011, Proceedings of the IEEE.

[20]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[21]  T. Warren Liao,et al.  Clustering of time series data - a survey , 2005, Pattern Recognit..

[22]  Ana Busic,et al.  Ancillary service to the grid from deferrable loads: The case for intelligent pool pumps in Florida , 2013, 52nd IEEE Conference on Decision and Control.

[23]  Michael Chertkov,et al.  A Markov Process Approach to Ensemble Control of Smart Buildings , 2019, 2019 IEEE Milan PowerTech.

[24]  Ana Busic,et al.  Ancillary Service to the Grid Using Intelligent Deferrable Loads , 2014, IEEE Transactions on Automatic Control.

[25]  Michael Chertkov,et al.  Ensemble Control of Cycling Energy Loads: Markov Decision Approach , 2017, ArXiv.

[26]  Vicenç Gómez,et al.  Hierarchical Linearly-Solvable Markov Decision Problems , 2016, ICAPS.

[27]  Emanuel Todorov,et al.  Linearly-solvable Markov decision problems , 2006, NIPS.

[28]  Panajotis Agathoklis,et al.  A new thermostat for real-time price demand response: Cost, comfort and energy impacts of discrete-time control without deadband , 2015 .