3. Introduction to Dynamic Programming

This book concerns the use of a method known as dynamic programming (DP) to solve large classes of optimization problems. We will focus on discrete optimization problems for which a set or sequence of decisions must be made to optimize (minimize or maximize) some function of the decisions. There are of course numerous methods to solve discrete optimization problems, many of which are collectively known as mathematical programming methods. Our objective here is not to compare these other mathematical programming methods with dynamic programming. Each has advantages and disadvantages, as discussed in many other places. However, we will note that the most prominent of these other methods is linear programming. As its name suggests, it has limitations associated with its linearity assumptions whereas many problems are nonlinear. Nevertheless, linear programming and its variants and extensions (some that allow nonlinearities) have been used to solve many real world problems, in part because very early in its development software tools (based on the simplex method) were made available to solve linear programming problems. On the other hand, no such tools have been available for the much more general method of dynamic programming, largely due to its very generality. One of the objectives of this book is to describe a software tool for solving dynamic programming problems that is general, practical, and easy to use, certainly relative to any of the other tools that have appeared from time to time. One reason that simplex-based tools for solving linear programming problems have been successful is that, by the nature of linear programming, problem specification is relatively easy. A basic LP problem can be specified essentially as a system or matrix of equations with a finite set of numerical variables as unknowns. That is, the input to an LP software tool can be provided in a tabular form, known as a tableaux. This also makes it easy to formulate LP problems as a spreadsheet. This led to spreadsheet system providers to include in their product an LP solver, as is the case with Excel. A software tool for solving dynamic programming problems is much more difficult to design, in part because the problem specification task in itself