Energy Storage Arbitrage in Real-Time Markets via Reinforcement Learning

In this paper, we derive a temporal arbitrage policy for storage via reinforcement learning. Real-time price arbitrage is an important source of revenue for storage units, but designing good strategies have proven to be difficult because of the highly uncertain nature of the prices. Instead of current model predictive or dynamic programming approaches, we use reinforcement learning to design an optimal arbitrage policy. This policy is learned through repeated charge and discharge actions performed by the storage unit through updating a value matrix. We design a reward function that does not only reflect the instant profit of charge/discharge decisions but also incorporate the history information. Simulation results demonstrate that our designed reward function leads to significant performance improvement compared with existing algorithms.

[1]  R. Byrne Estimating the Maximum Potential Revenue for Grid Connected Electricity Storage: Arbitrage and Regulation , 2012 .

[2]  Warren B. Powell,et al.  Optimal Hour-Ahead Bidding in the Real-Time Electricity Market with Battery Storage Using Approximate Dynamic Programming , 2014, INFORMS J. Comput..

[3]  H. Vincent Poor,et al.  Scheduling Power Consumption With Price Uncertainty , 2011, IEEE Transactions on Smart Grid.

[4]  S. Borenstein The Long-Run Efficiency of Real-Time Electricity Pricing , 2005 .

[5]  R. Weron A look into the future of electricity (spot) price forecasting , 2014 .

[6]  Junjie Qin,et al.  Online Modified Greedy algorithm for storage control under uncertainty , 2016, 2016 IEEE Power and Energy Society General Meeting (PESGM).

[7]  Sandia Report,et al.  Energy Storage for the Electricity Grid: Benefits and Market Potential Assessment Guide A Study for the DOE Energy Storage Systems Program , 2010 .

[8]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[9]  J. Apt,et al.  Economics of electric energy storage for energy arbitrage and regulation in New York , 2007 .

[10]  Zhi Zhou,et al.  Energy Storage Arbitrage Under Day-Ahead and Real-Time Price Uncertainty , 2018, IEEE Transactions on Power Systems.

[11]  Peter Dayan,et al.  Q-learning , 1992, Machine Learning.

[12]  Kyle Bradbury,et al.  Economic viability of energy storage systems based on price arbitrage potential in real-time U.S. electricity markets , 2014 .

[13]  G. Baiocchi,et al.  The value of arbitrage for energy storage: Evidence from European electricity markets , 2016 .

[14]  R. Bellman,et al.  Dynamic Programming and Markov Processes , 1960 .

[15]  P. Denholm,et al.  Estimating the value of electricity storage in PJM: Arbitrage and some welfare effects , 2009 .

[16]  M. Sandiford,et al.  Estimating the value of electricity storage in an energy-only wholesale market , 2015 .

[17]  Chi-Keung Woo,et al.  The impact of wind generation on the electricity spot-market price level and variance: The Texas experience , 2011 .