Approximate dynamic programming in transportation and logistics: a unified framework

Deterministic optimization has enjoyed a rich place in transportation and logistics, where it represents a mature field with established modeling and algorithmic strategies. By contrast, sequential stochastic optimization models (dynamic programs) have been plagued by the lack of a common modeling framework, and by algorithmic strategies that just do not seem to scale to real-world problems in transportation. This paper is designed as a tutorial of the modeling and algorithmic framework of approximate dynamic programming; however, our perspective on approximate dynamic programming is relatively new, and the approach is new to the transportation research community. We present a simple yet precise modeling framework that makes it possible to integrate most algorithmic strategies into four fundamental classes of policies, the design of which represents approximate solutions to these dynamic programs. The paper then uses problems in transportation and logistics to indicate settings in which each of the four classes of policies represents a natural solution strategy, highlighting the fact that the design of effective policies for these complex problems will remain an exciting area of research for many years. Along the way, we provide a link between dynamic programming, stochastic programming and stochastic search.

[1]  Peter Kall,et al.  Stochastic Programming , 1995 .

[2]  G. Dantzig,et al.  The Allocation of Aircraft to Routes—An Example of Linear Programming Under Uncertain Demand , 1956 .

[3]  Warren B. Powell,et al.  Optimal Learning , 2022, Encyclopedia of Machine Learning and Data Mining.

[4]  Teodor Gabriel Crainic,et al.  A cooperative parallel meta-heuristic for the vehicle routing problem with time windows , 2005, Comput. Oper. Res..

[5]  Gilbert Laporte,et al.  Solving a Dynamic and Stochastic Vehicle Routing Problem with a Sample Scenario Hedging Heuristic , 2006, Transp. Sci..

[6]  William T. Ziemba,et al.  Applications of Stochastic Programming , 2005 .

[7]  Warren B. Powell,et al.  Stochastic and dynamic networks and routing , 1995 .

[8]  Russell Bent,et al.  Online stochastic combinatorial optimization , 2006 .

[9]  Warren B. Powell,et al.  An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application , 2009, Transp. Sci..

[10]  Warren B. Powell,et al.  Strategic, Tactical and Real-Time Planning of Locomotives at Norfolk Southern Using Approximate Dynamic Programming , 2012 .

[11]  Pascal Van Hentenryck,et al.  An anytime multistep anticipatory algorithm for online stochastic combinatorial optimization , 2011, Ann. Oper. Res..

[12]  Pascal Van Hentenryck,et al.  Amsaa: A Multistep Anticipatory Algorithm for Online Stochastic Combinatorial Optimization , 2008, CPAIOR.

[13]  Warren B. Powell,et al.  A Representational Paradigm for Dynamic Resource Transformation Problems , 2001, Ann. Oper. Res..

[14]  Dean A. Jones,et al.  Robust optimization for fleet planning under uncertainty , 2003 .

[15]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.

[16]  John R. Birge,et al.  Introduction to Stochastic Programming , 1997 .

[17]  Panos M. Pardalos,et al.  Approximate dynamic programming: solving the curses of dimensionality , 2009, Optim. Methods Softw..

[18]  Werner Römisch,et al.  Scenario tree modeling for multistage stochastic programs , 2009, Math. Program..

[19]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[20]  Gilbert Laporte,et al.  An Exact Algorithm for the Vehicle Routing Problem with Stochastic Demands and Customers , 1995, Transp. Sci..

[21]  Warrren B Powell,et al.  Value Function Approximation using Multiple Aggregation for Multiattribute Resource Management , 2008 .

[22]  Russell Bent,et al.  Scenario-Based Planning for Partially Dynamic Vehicle Routing with Stochastic Customers , 2004, Oper. Res..

[23]  Warren B. Powell,et al.  Approximate Dynamic Programming Captures Fleet Operations for Schneider National , 2010, Interfaces.

[24]  B PowellWarren,et al.  An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, II , 2002 .

[25]  Julia L. Higle,et al.  Stochastic Decomposition: A Statistical Method for Large Scale Stochastic Linear Programming , 1996 .

[26]  Julia L. Higle,et al.  Stochastic Decomposition: An Algorithm for Two-Stage Linear Programs with Recourse , 1991, Math. Oper. Res..

[27]  C. N Bouza,et al.  Spall, J.C. Introduction to stochastic search and optimization. Estimation, simulation and control. Wiley Interscience Series in Discrete Mathematics and Optimization, 2003 , 2004 .

[28]  Warren B. Powell,et al.  An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, II: Multiperiod Travel Times , 2002, Transp. Sci..

[29]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[30]  Warrren B Powell,et al.  An Adaptive, Distribution-Free Algorithm for the Newsvendor Problem with Censored Demands, with Applications to Inventory and Distribution , 2001 .

[31]  U. Rieder,et al.  Markov Decision Processes , 2010 .

[32]  Jean-Yves Potvin,et al.  Vehicle routing and scheduling with dynamic travel times , 2006, Comput. Oper. Res..

[33]  Andrew G. Barto,et al.  Learning to Act Using Real-Time Dynamic Programming , 1995, Artif. Intell..

[34]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[35]  Alexander Shapiro,et al.  Lectures on Stochastic Programming: Modeling and Theory , 2009 .

[36]  Warren B. Powell,et al.  Approximate dynamic programming for management of high‐value spare parts , 2009, Journal of Manufacturing Technology Management.

[37]  Jesfis Peral,et al.  Heuristics -- intelligent search strategies for computer problem solving , 1984 .

[38]  Warren B. Powell,et al.  Optimal Learning: Powell/Optimal , 2012 .

[39]  Richard F. Hartl,et al.  Metaheuristics for the dynamic stochastic dial-a-ride problem with expected return transports , 2011, Comput. Oper. Res..

[40]  Loo Hay Lee,et al.  An approximate dynamic programming approach for the empty container allocation problem , 2007 .

[41]  R. Young,et al.  Economies of Scale , 1987 .

[42]  David Sellers,et al.  North American freight rail industry real-time optimized equipment distribution systems: State of the practice , 2011 .

[43]  John N. Tsitsiklis,et al.  Feature-based methods for large scale dynamic programming , 2004, Machine Learning.

[44]  John N. Tsitsiklis,et al.  Asynchronous Stochastic Approximation and Q-Learning , 1994, Machine Learning.

[45]  Michel Gendreau,et al.  A Tabu Search Heuristic for the Vehicle Routing Problem with Soft Time Windows , 1997, Transp. Sci..

[46]  Warren B. Powell,et al.  Learning Algorithms for Separable Approximations of Discrete Stochastic Optimization Problems , 2004, Math. Oper. Res..

[47]  Michel Gendreau,et al.  Dynamic and Stochastic Models for the Allocation of Empty Containers , 1993, Oper. Res..

[48]  William C. Jordan,et al.  A STOCHASTIC, DYNAMIC NETWORK MODEL FOR RAILROAD CAR DISTRIBUTION , 1983 .

[49]  Teodor Gabriel Crainic,et al.  Planning models for freight transportation , 1997 .

[50]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[51]  Michel Gendreau,et al.  Parallel Tabu Search for Real-Time Vehicle Routing and Dispatching , 1999, Transp. Sci..

[52]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[53]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[54]  Dimitri P. Bertsekas,et al.  Rollout Algorithms for Stochastic Scheduling Problems , 1999, J. Heuristics.

[55]  Russell Bent,et al.  Online stochastic optimization under time constraints , 2005, Ann. Oper. Res..

[56]  Warren B. Powell,et al.  SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy , 2012, INFORMS J. Comput..

[57]  Judea Pearl,et al.  Heuristics : intelligent search strategies for computer problem solving , 1984 .

[58]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[59]  George B. Dantzig,et al.  The Problem of Routing Aircraft--A Mathematical Solution. , 1954 .

[60]  Warren B. Powell,et al.  Hierarchical Knowledge Gradient for Sequential Sampling , 2011, J. Mach. Learn. Res..

[61]  Allan Larsen,et al.  The Dynamic Vehicle Routing Problem , 2000 .

[62]  Christos D. Tarantilis,et al.  Dynamic Vehicle Routing Problems , 2014, Vehicle Routing.

[63]  Warren B. Powell,et al.  An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, I: Single Period Travel Times , 2002, Transp. Sci..

[64]  R. Tyrrell Rockafellar,et al.  Scenarios and Policy Aggregation in Optimization Under Uncertainty , 1991, Math. Oper. Res..

[65]  Warren B. Powell,et al.  Dynamic-Programming Approximations for Stochastic Time-Staged Integer Multicommodity-Flow Problems , 2006, INFORMS J. Comput..

[66]  O. H. Brownlee,et al.  ACTIVITY ANALYSIS OF PRODUCTION AND ALLOCATION , 1952 .

[67]  Gilbert Laporte,et al.  An Integer L-Shaped Algorithm for the Capacitated Vehicle Routing Problem with Stochastic Demands , 2002, Oper. Res..

[68]  Michel Gendreau,et al.  Exploiting Knowledge About Future Demands for Real-Time Vehicle Dispatching , 2006, Transp. Sci..

[69]  Paolo Toth,et al.  The Vehicle Routing Problem , 2002, SIAM monographs on discrete mathematics and applications.

[70]  Michel Gendreau,et al.  Vehicle Routing Problem with Time Windows, Part I: Route Construction and Local Search Algorithms , 2005, Transp. Sci..

[71]  Michel Gendreau,et al.  Economies of Scale in Empty Freight Car Distribution in Scheduled Railways , 2004, Transp. Sci..

[72]  Michel Gendreau,et al.  A review of dynamic vehicle routing problems , 2013, Eur. J. Oper. Res..