Benchmarking a Scalable Approximate Dynamic Programming Algorithm for Stochastic Control of Grid-Level Energy Storage

We present and benchmark an approximate dynamic programming algorithm that is capable of designing near-optimal control policies for a portfolio of heterogenous storage devices in a time-dependent environment, where wind supply, demand, and electricity prices may evolve stochastically. We found that the algorithm was able to design storage policies that are within 0.08% of optimal on deterministic models, and within 0.86% on stochastic models. We use the algorithm to analyze a dual-storage system with different capacities and losses, and show that the policy properly uses the low-loss device (which is typically much more expensive) for high-frequency variations. We close by demonstrating the algorithm on a five-device system. The algorithm easily scales to handle heterogeneous portfolios of storage devices distributed over the grid and more complex storage networks. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0768.

[1]  Panos M. Pardalos,et al.  Approximate dynamic programming: solving the curses of dimensionality , 2009, Optim. Methods Softw..

[2]  Vincenzo Marano,et al.  A stochastic dynamic programming model for co-optimization of distributed energy storage , 2013, Energy Systems.

[3]  Stein-Erik Fleten,et al.  Stochastic programming for optimizing bidding strategies of a Nordic hydropower producer , 2007, Eur. J. Oper. Res..

[4]  Warren B. Powell,et al.  A Distributed Decision-Making Structure for Dynamic Resource Allocation Using Nonlinear Functional Approximations , 2005, Oper. Res..

[5]  G. Kariniotakis,et al.  Management of Energy Storage Coordinated with Wind Power under Electricity Market Conditions , 2008, Proceedings of the 10th International Conference on Probablistic Methods Applied to Power Systems.

[6]  Hamidreza Zareipour,et al.  Energy storage for mitigating the variability of renewable electricity sources: An updated review , 2010 .

[7]  Subhashish Bhattacharya,et al.  Rule-Based Control of Battery Energy Storage for Dispatching Intermittent Renewable Sources , 2010, IEEE Transactions on Sustainable Energy.

[8]  Srdjan M. Lukic,et al.  Energy Storage Systems for Transport and Grid Applications , 2010, IEEE Transactions on Industrial Electronics.

[9]  Paul Denholm,et al.  Benefits of Colocating Concentrating Solar Power and Wind , 2013, IEEE Transactions on Sustainable Energy.

[10]  Arild Helseth,et al.  Hydrothermal scheduling in Norway using stochastic dual dynamic programming; a large-scale case study , 2015, 2015 IEEE Eindhoven PowerTech.

[11]  Goran Strbac,et al.  Value of storage in providing balancing services for electricity generation systems with high wind penetration , 2006 .

[12]  M.V.F. Pereira,et al.  Stochastic optimization of transmission constrained and large scale hydrothermal systems in a competitive framework , 2003, 2003 IEEE Power Engineering Society General Meeting (IEEE Cat. No.03CH37491).

[13]  Vitor L. de Matos,et al.  Dynamic sampling algorithms for multi-stage stochastic programs with risk aversion , 2012, Eur. J. Oper. Res..

[14]  Warren B. Powell,et al.  Dynamic-Programming Approximations for Stochastic Time-Staged Integer Multicommodity-Flow Problems , 2006, INFORMS J. Comput..

[15]  Warren B. Powell,et al.  Learning Algorithms for Separable Approximations of Discrete Stochastic Optimization Problems , 2004, Math. Oper. Res..

[16]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[17]  S. Minner,et al.  Optimal day-ahead trading and storage of renewable energies—an approximate dynamic programming approach , 2010 .

[18]  Ramteen Sioshansi,et al.  Market protocols in ERCOT and their effect on wind generation , 2010 .

[19]  Warrren B Powell,et al.  An Adaptive, Distribution-Free Algorithm for the Newsvendor Problem with Censored Demands, with Applications to Inventory and Distribution , 2001 .

[20]  Warren B. Powell,et al.  Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming , 2006, Machine Learning.

[21]  G. Pritchard,et al.  Hydroelectric reservoir optimization in a pool market , 2005, Math. Program..

[22]  J.A.P. Lopes,et al.  On the optimization of the daily operation of a wind-hydro power plant , 2004, IEEE Transactions on Power Systems.

[23]  J.P. Barton,et al.  Energy storage and its use with intermittent renewable energy , 2004, IEEE Transactions on Energy Conversion.

[24]  Richard S. Sutton,et al.  A Menu of Designs for Reinforcement Learning Over Time , 1995 .

[25]  Stefan Minner,et al.  Optimizing Trading Decisions for Hydro Storage Systems Using Approximate Dual Dynamic Programming , 2013, Oper. Res..

[26]  P. J. Werbos,et al.  Backpropagation and neurocontrol: a review and prospectus , 1989, International 1989 Joint Conference on Neural Networks.

[27]  Alon Kuperman,et al.  Battery–ultracapacitor hybrids for pulsed current loads: A review , 2011 .

[28]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[29]  Alexander Shapiro,et al.  Risk neutral and risk averse Stochastic Dual Dynamic Programming method , 2013, Eur. J. Oper. Res..

[30]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[31]  Warren B. Powell,et al.  An Optimal Approximate Dynamic Programming Algorithm for Concave, Scalar Storage Problems With Vector-Valued Controls , 2013, IEEE Transactions on Automatic Control.

[32]  J. Iannucci,et al.  Energy Storage Benefits and Market Analysis Handbook A Study for the DOE Energy Storage Systems Program , 2004 .

[33]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[34]  Haosheng Zhou.  New energy storage devices for post lithium-ion batteries , 2013 .

[35]  Sonia Martínez,et al.  Storage Size Determination for Grid-Connected Photovoltaic Systems , 2011, IEEE Transactions on Sustainable Energy.

[36]  Pedram Mokrian,et al.  A Stochastic Programming Framework for the Valuation of Electricity Storage , 2022 .

[37]  Warren B. Powell,et al.  SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy , 2012, INFORMS J. Comput..

[38]  M. Trovato,et al.  Planning and Operating Combined Wind-Storage System in Electricity Market , 2012, IEEE Transactions on Sustainable Energy.

[39]  Nicola Secomandi,et al.  Is it More Valuable to Store or Destroy Electricity Surpluses? , 2012 .

[40]  Alfons Vervaet,et al.  Lead-acid battery model for the derivation of Peukert’s law , 1999 .

[41]  Benjamin Van Roy,et al.  A neuro-dynamic programming approach to retailer inventory management , 1997, Proceedings of the 36th IEEE Conference on Decision and Control.