Temporal Dierences-Based Policy Iteration and Applications in Neuro-Dynamic Programming 1

We introduce a new policy iteration method for dynamic programming problems with discounted and undiscounted cost. The method is based on the notion of temporal dierences, and is primarily geared to the case of large and complex problems where the use of approximations is essential. We develop the theory of the method without approximation, we describe how to embed it within a neuro-dynamic programming/reinforcement learning context where feature-based approximation architectures are used, we relate it to TD( ) methods, and we illustrate its use in the training of a tetris playing program.