Scenario Trees and Policy Selection for Multistage Stochastic Programming Using Machine Learning

In the context of multistage stochastic optimization problems, we propose a hybrid strategy for generalizing to nonlinear decision rules, using machine learning, a finite data set of constrained vector-valued recourse decisions optimized using scenario-tree techniques from multistage stochastic programming. The decision rules are based on a statistical model inferred from a given scenario-tree solution and are selected by out-of-sample simulation given the true problem. Because the learned rules depend on the given scenario tree, we repeat the procedure for a large number of randomly generated scenario trees and then select the best solution policy found for the true problem. The scheme leads to an ex post selection of the scenario tree itself. Numerical tests evaluate the dependence of the approach on the machine learning aspects and show cases where one can obtain near-optimal solutions, starting with a “weak” scenario-tree generator that randomizes the branching structure of the trees.

[1]  D. Ernst,et al.  Multistage Stochastic Programming: A Scenario Tree Based Approach to Planning under Uncertainty , 2011 .

[2]  David P. Morton,et al.  Monte Carlo bounding techniques for determining solution quality in stochastic programs , 1999, Oper. Res. Lett..

[3]  G. Pflug,et al.  Quantitative Fund Management , 2008 .

[4]  Kai Huang,et al.  The Value of Multistage Stochastic Programming in Capacity Planning Under Uncertainty , 2009, Oper. Res..

[5]  Jitka Dupacová,et al.  Scenarios for Multistage Stochastic Programs , 2000, Ann. Oper. Res..

[6]  John R. Birge,et al.  Introduction to Stochastic Programming , 1997 .

[7]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[8]  Michael H. Bowling,et al.  Apprenticeship learning using linear programming , 2008, ICML '08.

[9]  Anukal Chiralaksanakul.  Monte Carlo methods for multi-stage stochastic programs , 2003 .

[10]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[11]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[12]  Stein W. Wallace,et al.  Generating Scenario Trees for Multistage Decision Problems , 2001, Manag. Sci..

[13]  Bart De Schutter,et al.  Reinforcement Learning and Dynamic Programming Using Function Approximators , 2010 .

[14]  Alexander Shapiro,et al.  Inference of statistical bounds for multistage stochastic programming problems , 2003, Math. Methods Oper. Res..

[15]  William T. Ziemba,et al.  Applications of Stochastic Programming , 2005 .

[16]  Boris Defourny,et al.  Machine Learning Solution Methods for Multistage Stochastic Programming , 2010 .

[17]  A. O'Hagan,et al.  Curve Fitting and Optimal Design for Prediction , 1978 .

[18]  Warren B. Powell,et al.  “Approximate dynamic programming: Solving the curses of dimensionality” by Warren B. Powell , 2007, Wiley Series in Probability and Statistics.

[19]  Pieter Abbeel,et al.  Learning for control from multiple demonstrations , 2008, ICML '08.

[20]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[21]  Panos M. Pardalos,et al.  Optimization in the Energy Industry , 2009 .

[22]  Teemu Pennanen,et al.  Epi-Convergent Discretizations of Multistage Stochastic Programs , 2005, Math. Oper. Res..

[23]  Gilles Pagès,et al.  Optimal quadratic quantization for numerics: the Gaussian case , 2003, Monte Carlo Methods Appl..

[24]  Werner Römisch,et al.  Scenario tree modeling for multistage stochastic programs , 2009, Math. Program..

[25]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[26]  Pieter Abbeel,et al.  Apprenticeship learning via inverse reinforcement learning , 2004, ICML.

[27]  W. Römisch,et al.  Stability and Scenario Trees for Multistage Stochastic Programs , 2010 .

[28]  Jean-Philippe Vial,et al.  Step decision rules for multistage stochastic programming: A heuristic approach , 2008, Autom..

[29]  Woo Chang Kim,et al.  Multistage Financial Planning Models: Integrating Stochastic Programs and Policy Simulators , 2010 .

[30]  Teemu Pennanen,et al.  Numerical study of discretizations of multistage stochastic programs , 2008, Kybernetika.

[31]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[32]  R. Tyrrell Rockafellar,et al.  Scenarios and Policy Aggregation in Optimization Under Uncertainty , 1991, Math. Oper. Res..

[33]  Roy Kouwenberg,et al.  Scenario generation and stochastic programming models for asset liability management , 2001, Eur. J. Oper. Res..

[34]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[35]  Yurii Nesterov,et al.  Confidence level solutions for stochastic programming , 2000, Autom..

[36]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[37]  Teemu Pennanen,et al.  Epi-convergent discretizations of multistage stochastic programs via integration quadratures , 2008, Math. Program..

[38]  Louis Wehenkel,et al.  Bounds for Multistage Stochastic Programs Using Supervised Learning Strategies , 2009, SAGA.

[39]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[40]  John R. Birge,et al.  Introduction to Stochastic programming (2nd edition), Springer verlag, New York , 2011 .

[41]  Karl Frauendorfer,et al.  Barycentric scenario trees in convex multistage stochastic programming , 1996, Math. Program..

[42]  Csaba Szepesvári,et al.  Algorithms for Reinforcement Learning , 2010, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[43]  Stefan Vigerske,et al.  Numerical evaluation of approximation methods in stochastic programming , 2010 .

[44]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.