From Infinite to Finite Programs: Explicit Error Bounds with Applications to Approximate Dynamic Programming

We consider linear programming (LP) problems in infinite dimensional spaces that are in general computationally intractable. Under suitable assumptions, we develop an approximation bridge from the infinite dimensional LP to tractable finite convex programs in which the performance of the approximation is quantified explicitly. To this end, we adopt the recent developments in two areas of randomized optimization and first-order methods, leading to a priori as well as a posteriori performance guarantees. We illustrate the generality and implications of our theoretical results in the special case of the long-run average cost and discounted cost optimal control problems in the context of Markov decision processes on Borel spaces. The applicability of the theoretical results is demonstrated through a fisheries management problem.

[1]  Tamás Linder,et al.  On the Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces , 2015, Math. Oper. Res..

[2]  Marco C. Campi,et al.  The Exact Feasibility of Randomized Solutions of Uncertain Convex Programs , 2008, SIAM J. Optim..

[3]  Moshe Ben-Horim,et al.  A linear programming approach , 1977 .

[4]  A. Nemirovski,et al.  Scenario Approximations of Chance Constraints , 2006 .

[5]  I. Csiszár.  $I$-Divergence Geometry of Probability Distributions and Minimization Problems , 1975 .

[6]  François Dufour,et al.  Finite Linear Programming Approximations of Constrained Discounted Markov Decision Processes , 2013, SIAM J. Control. Optim..

[7]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[8]  W. Fleming.  Book Review: Discrete-time Markov control processes: Basic optimality criteria , 1997 .

[9]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[10]  O. Hernández-Lerma,et al.  Further topics on discrete-time Markov control processes , 1999 .

[11]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[12]  François Dufour,et al.  Stochastic approximations of constrained discounted Markov decision processes , 2014 .

[13]  John Lygeros,et al.  Motion Planning for Continuous-Time Stochastic Processes: A Dynamic Programming Approach , 2012, IEEE Transactions on Automatic Control.

[14]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[15]  Rida T. Farouki,et al.  The Bernstein polynomial basis: A centennial retrospective , 2012, Comput. Aided Geom. Des..

[16]  Mikhail Borisovich Nevelʹson,et al.  Stochastic Approximation and Recursive Estimation , 1976 .

[17]  M. Sion.  On general minimax theorems , 1958 .

[18]  S. Pearson.  Moments , 2020, Narrative inquiry in bioethics.

[19]  Franccois Dufour,et al.  A linear programming formulation for constrained discounted continuous control for piecewise deterministic Markov processes , 2014 .

[20]  Esimo Hern,et al.  CONSTRAINED AVERAGE COST MARKOV CONTROL PROCESSES IN BOREL SPACES , 2003 .

[21]  Benjamin Van Roy,et al.  The Linear Programming Approach to Approximate Dynamic Programming , 2003, Oper. Res..

[22]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[23]  J. Lasserre.  Moments, Positive Polynomials And Their Applications , 2009 .

[24]  Quanxin Zhu,et al.  Average optimality for Markov decision processes in borel spaces: a new condition and approach , 2006, Journal of Applied Probability.

[25]  John Lygeros,et al.  Approximate dynamic programming for stochastic reachability , 2013, 2013 European Control Conference (ECC).

[26]  Marco Pavone,et al.  Stochastic Optimal Control , 2015 .

[27]  Onésimo Hernández-Lerma,et al.  Approximation Schemes for Infinite Linear Programs , 1998, SIAM J. Optim..

[28]  J. Guermond,et al.  Theory and practice of finite elements , 2004 .

[29]  M. K rn,et al.  Stochastic Optimal Control , 1988 .

[30]  Xuan Vinh Doan,et al.  Approximating integrals of multivariate exponentials: A moment approach , 2008, Oper. Res. Lett..

[31]  Sheehan Olver,et al.  On the convergence rate of a modified Fourier series , 2009, Math. Comput..

[32]  J. Pach,et al.  Wiley‐Interscience Series in Discrete Mathematics and Optimization , 2011 .

[33]  John Lygeros,et al.  Approximation of Constrained Average Cost Markov Control Processes , 2014, 53rd IEEE Conference on Decision and Control.

[34]  O. Hernondex-lerma,et al.  Adaptive Markov Control Processes , 1989 .

[35]  C. Villani.  Topics in Optimal Transportation , 2003 .

[36]  Vivek S. Borkar,et al.  Stochastic Control with Imperfect Models , 2008, SIAM J. Control. Optim..

[37]  M. K. Ghosh,et al.  Discrete-time controlled Markov processes with average cost criterion: a survey , 1993 .

[38]  John N. Tsitsiklis,et al.  Analysis of temporal-difference learning with function approximation , 1996, NIPS 1996.

[39]  John Lygeros,et al.  The stochastic reach-avoid problem and set characterization for diffusions , 2012, Autom..

[40]  Jean-Philippe Vial,et al.  Robust Optimization , 2021, ICORES.

[41]  F. Dufour,et al.  Approximation of average cost Markov decision processes using empirical distributions and concentration inequalities , 2015 .

[42]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[43]  John Lygeros,et al.  Performance Bounds for the Scenario Approach and an Extension to a Class of Non-Convex Programs , 2013, IEEE Transactions on Automatic Control.

[44]  D. Bertsekas.  Convergence of discretization procedures in dynamic programming , 1975 .

[45]  Gautam Appa,et al.  Linear Programming in Infinite-Dimensional Spaces , 1989 .

[46]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[47]  S. Y. Wu,et al.  Extremal points and optimal solutions for general capacity problems , 1992, Math. Program..

[48]  F. Dufour,et al.  Approximation of Markov decision processes with general state space , 2012 .

[49]  Benjamin Van Roy,et al.  On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming , 2004, Math. Oper. Res..

[50]  Sean P. Meyn,et al.  Rationally Inattentive Control of Markov Processes , 2015, SIAM J. Control. Optim..

[51]  Tamás Linder,et al.  Asymptotic Optimality of Finite Approximations to Markov Decision Processes with General State and Action Spaces , 2015, ArXiv.

[52]  John N. Tsitsiklis,et al.  Actor-Critic Algorithms , 1999, NIPS.

[53]  Maria Prandini,et al.  Stochastic Reachability: Theory and Numerical Approximation , 2006 .