Balancing Imbalance: On using reinforcement learning to increase stability in smart electricity grids

Over the years interest in renewable energy sources is growing and the amount of electricity provided by such resources, such as photovoltaic (PV) or wind energy is increasing. This results in a number of changes in the way that electricity is both produced and consumed in society. The supply of renewable energy sources is highly dependent on environmental changes and therefore hard to predict and adjust. As a response new methods have been proposed to shift this control to the demand side of the electricity grid by the means of Demand Side Management. Furthermore it can be seen that there is a shift from a situation in which the supply of electricity is managed by a small group of very large suppliers to a larger group of smaller suppliers. This, among others because of the increase of for example wind farms and the opportunity for residentials to generate their own power (by, for example, installing PV panels). One of the responsibilities of the larger producers is to maintain the balance between demand and supply across the electricity grid. This means that these parties should be able to ramp up or ramp down their production whenever the demand is higher or lower than expected in different regions. These adjustments happen on the so-called balancing market. Due to minimum production rates these markets are only open to larger suppliers and not to the group of smaller prosumers (parties that both consume and produce electricity). However, as this group becomes larger this group would be a great addition to the electricity markets when it comes to maintaining stability between the supply and demand in the electricity grid. One solution to add these smaller parties to the electricity grid is to bundle groups of small prosumers (a party that both produces and consumes electricity) into a cluster, controlled by a so-called aggregator. The aggregator can then offer a certain range of power within which it is able to ramp the amount of consumption (or production) up or down. However, ramping up or down too much at a given moment, might result in an imbalance in the opposite direction at a later point in time. This means that any imbalances that might be restored by the aggregator initially might come back even more extreme in the near future. In this thesis it is shown how reinforcement learning can be applied to successfully learn the boundary conditions within which an aggregator can safely ramp up or down its consumption. Furthermore Neural Fitted CACLA (NFCACLA) is proposed as a new reinforcement learning algorithm, which is a neural fitted variant on the existing CACLA algorithm.