Feature Selection for Neuro-Dynamic Programming

AbstractNeuro-Dynamic Programming encompasses techniques from both reinforcement learn-ing and approximate dynamic programming. Feature selection refers to the choice of basisthat de nes the function class that is required in the application of these techniques.This chapter reviews two popular approaches to neuro-dynamic programming, TD-learning and Q-learning. The main goal of the chapter is to demonstrate how insightfrom idealized models can be used as a guide for feature selection for these algorithms.Several approaches are surveyed, including uid and di usion models, and the applicationof idealized models arising from mean- eld game approximations. The theory is illustratedwith several examples.Keywords: Optimal Control, Stochastic Control, Approximate Dynamic Programming,Reinforcement Learning2000 AMS Subject Classi cation: 49L20, 93E20, 93E35, 60J10 1 Introduction If you have taken a course that mentioned a Riccati equation, then you have already beenexposed to approximate dynamic programming: No physical system is linear! The modelis assumed to be linear, and the cost function is assumed quadratic, so that a closed-formexpression for the optimal feedback law can be computed.In many cases, a linear approximation is not easily justi ed. In particular, in applicationsfrom operations research or chemical engineering, the state space is usually constrained. Toobtain an e ective feedback law for control, we must nd alternative approaches to approxi-mation.Techniques from approximate dynamic programming and reinforcement learning are de-signed to obtain such approximations [1, 18]. In particular, in TD-learning and relatedapproaches, there is no attempt to approximate the system or cost function. Instead, thesolution to a dynamic programming equation is approximated directly, within a prescribed nite-dimensional function class. A key determinant of the success of these techniques is theselection of this function class, also known as the basis.