Alternative theoretical frameworks for finite horizon discrete-time stochastic optimal control

This talk will provide highlights of recent research aimed at constructing a unifying and mathematically rigorous theory of dynamic programming and stochastic optimal control. Stochastic optimal control problems are usually analyzed under one of three types of assumptions: a) Countability assumptions on the underlying probability space - this eliminates all difficulties of measure theoretic nature, b) Semicontinuity assumptions under which the existence of Borel measurable policies can be guaranteed (Maitra, Sh�, Freedman), and c) Borel measurability assumptions under which the existence of p-optimal or p-?-optimal Borel measurable policies can be guaranteed (Blackwell, Strauch). We introduce a general theoretical framework based on outer integration which contains these three models as special cases. Within this framework all known results for finite horizon problems together with some new ones are proved and subsequently specialized. The long-standing measure theoretic difficulties in the Borel space model of Blackwell and Strauch are traced to inadequacies in the class of Borel measurable policies. An important new feature of our specialization to this model is the expansion of the class of admissible policies to include all universally measurable policies. We show that everywhere optimal or nearly optimal policies exist within this class and this enables us to dispense with the notion of p-optimality. Furthermore by utilizing the extra flexibility afforded by universally measurable policies we are able to provide for the first time a satisfactory and mathematically rigorous treatment of problems of imperfect state information using sufficient statistics. The measurability properties which a satisfactory class of policies must have are also delineated. The minimal class of policies with these properties is constructed and shown to lie strictly between the classes of Borel measurable and universally measurable policies. A detailed treatment appears in papers available from the authors on request, as well as in the authors' monograph Stochastic Optimal Control: The Discrete-Time Case, to be published by Academic Press in the summer of 1978.

[1]  J. Neumann On Rings of Operators. Reduction Theory , 1949 .

[2]  D. Blackwell Discounted Dynamic Programming , 1965 .

[3]  P. Meyer Probability and potentials , 1966 .

[4]  K. Hinderer,et al.  Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter , 1970 .

[5]  Claude Dellacherie Ensembles analytiques, capacités, mesures de Hausdorff , 1972 .

[6]  L. Brown,et al.  Measurable Selections of Extrema , 1973 .

[7]  D. Blackwell,et al.  The Optimal Reward Operator in Dynamic Programming , 1974 .

[8]  D. Freedman The Optimal Reward Operator in Special Classes of Dynamic Programming Problems , 1974 .

[9]  M. Schäl On dynamic programming: Compactness of the space of policies , 1975 .

[10]  Manfred SchÄl,et al.  Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal , 1975 .

[11]  Paul Olsen Multistage Stochastic Programming with Recourse: The Equivalent Deterministic Problem , 1976 .

[12]  Paul Olsen When Is a Multistage Stochastic Programming Problem Well-Defined? , 1976 .

[13]  Paul Olsen Multistage Stochastic Programming with Recourse As Mathematical Programming in an $L_p $ Space , 1976 .

[14]  R. Rockafellar,et al.  Stochastic convex programming: basic duality. , 1976 .

[15]  NEW THEORETICAL FRAMEWORK FOR FINITE HORIZON STOCHASTIC CONTROL. , 1976 .

[16]  R. Rockafellar,et al.  Stochastic Convex Programming: Relatively Complete Recourse and Induced Feasibility , 1976 .

[17]  D. Bertsekas,et al.  Equivalent stochastic and deterministic optimal control problems , 1976, 1976 IEEE Conference on Decision and Control including the 15th Symposium on Adaptive Processes.

[18]  S. Shreve Dynamic Programming in Complete Separable Spaces , 1977 .

[19]  C.C. White,et al.  Dynamic programming and stochastic control , 1978, Proceedings of the IEEE.

[20]  S. Shreve Probability measures and the C-sets of Selivanovskij , 1978 .

[21]  O. Gaans Probability measures on metric spaces , 2022 .