Deep Reinforcement Learning for Modeling Market-Oriented Grid User Behavior in Active Distribution Grids

Today's electrical grids frequently face congestion situations due to a combination of the increasing installation of renewable energy sources, the impact of sector coupling, and the delayed grid expansion. To lower associated grid expansion costs, load and generation peaks can be reduced with grid-supporting measures, such as energy flexibility. In the future, with market-based redispatch procurement, this flexibility could be offered on a local flexibility market. However, such flexibility procurements require an in-depth knowledge and replication of grid user behavior. For this purpose, we model the market-oriented grid user behavior in a two-stage market model for active distribution grids: the zonal electricity market and the local flexibility market. The methodology comprises a Markov decision process that is solved by a deep reinforcement learning algorithm. We compare our implementation to an optimization algorithm, which iteratively optimizes the zonal and local flexibility market bidding. The analysis of the resulting market-oriented grid user behavior shows an adjustment in the bidding strategy of market-participants in presence of the local flexibility market. Further, the results indicate that the presented model is capable of exploring gaming strategies that maximize the reward on both markets.