Dynamic Programming Equations

In this chapter we define many of the standard control problems whose numerical solutions will concern us in the subsequent chapters. Other, less familiar control problems will be discussed separately in later chapters. We will first define cost functionals for uncontrolled processes, and then formally discuss the partial differential equations which they satisfy. Then the cost functionals for the controlled problems will be stated and the partial differential equations for the optimal cost formally derived. These partial differential equations are generally known as Bellman equations or dynamic programming equations. The main tool in the derivations is Ito’s formula.