From robust model predictive control to stochastic optimal control and approximate dynamic programming: A perspective gained from a personal journey

Abstract Developments in robust model predictive control are reviewed from a perspective gained through a personal involvement in the research area during the past two decades. Various min–max MPC formulations are discussed in the setting of optimizing the “worst-case” performance in closed loop. One of the insights gained is that the conventional open-loop formulation of MPC is fundamentally flawed to address optimal control of systems with uncertain parameters, though it can be tailored to give conservative solutions with robust stability guarantees for special classes of problems. Dynamic programming (DP) may be the only general framework for obtaining closed-loop optimal control solutions for such systems. Due to the “curse of dimensionality (COD),” however, exact solution of DP is seldom possible. Approximate dynamic programming (ADP), which attempts to overcome the COD, is discussed with potential extensions and future challenges.

[1]  T. Badgwell.  Robust model predictive control of stable linear systems , 1997 .

[2]  Jay H. Lee,et al.  Choice of approximator and design of penalty function for an approximate dynamic programming based control approach , 2006 .

[3]  Wee Chin Wong,et al.  Postdecision-State-Based Approximate Dynamic Programming for Robust Predictive Control of Constrained Stochastic Processes , 2011 .

[4]  Warren B. Powell,et al.  “Approximate dynamic programming: Solving the curses of dimensionality” by Warren B. Powell , 2007, Wiley Series in Probability and Statistics.

[5]  Andrew W. Moore,et al.  Generalization in Reinforcement Learning: Safely Approximating the Value Function , 1994, NIPS.

[6]  W. L. De Koning,et al.  Infinite horizon optimal control of linear discrete time systems with stochastic parameters , 1982, Autom..

[7]  J H Lee,et al.  Approximate dynamic programming approach for process control , 2010, ICCAS 2010.

[8]  Jay H. Lee,et al.  Min-max predictive control techniques for a linear state-space system with a bounded set of input matrices , 2000, Autom..

[9]  R. Bellman.  Dynamic programming. , 1957, Science.

[10]  Alberto Bemporad,et al.  Robust model predictive control: A survey , 1998, Robustness in Identification and Control.

[11]  Jay H. Lee,et al.  Tuning of model predictive controllers for robust performance , 1994 .

[12]  Richard S. Sutton,et al.  Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming , 1990, ML.

[13]  M. Morari,et al.  Robust Stability of Constrained Model Predictive Control , 1993, 1993 American Control Conference.

[14]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[15]  Manfred Morari,et al.  Robust constrained model predictive control using linear matrix inequalities , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[16]  S. Pliska,et al.  Risk-Sensitive Dynamic Asset Management , 1999 .

[17]  S. Dreyfus.  Some Types of Optimal Control of Stochastic Systems , 1964 .

[18]  Warren B. Powell,et al.  Handbook of Learning and Approximate Dynamic Programming , 2006, IEEE Transactions on Automatic Control.

[19]  Robert F. Stengel,et al.  Optimal Control and Estimation , 1994 .

[20]  Jong Min Lee,et al.  Value function-based approach to the scheduling of multiple controllers , 2008 .

[21]  R. Stengel.  Stochastic Optimal Control: Theory and Application , 1986 .

[22]  Jay H. Lee,et al.  Approximate dynamic programming-based approaches for input-output data-driven control of nonlinear processes , 2005, Autom..

[23]  David Q. Mayne,et al.  Constrained model predictive control: Stability and optimality , 2000, Autom..

[24]  Jong Min Lee,et al.  An approximate dynamic programming based approach to dual adaptive control , 2009 .

[25]  Jay H. Lee,et al.  Approximate dynamic programming based approach to process control and scheduling , 2006, Comput. Chem. Eng..

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Hussam Nosair,et al.  Min–max control using parametric approximate dynamic programming , 2010 .

[28]  Mayuresh V. Kothare,et al.  An e!cient o"-line formulation of robust model predictive control using linear matrix inequalities (cid:1) , 2003 .

[29]  Antonio Vicino,et al.  Optimal estimation theory for dynamic systems with set membership uncertainty: An overview , 1991, Autom..

[30]  Evanghelos Zafiriou,et al.  Robust Model Predictive Control of Processes with Hard Constraints. , 1990 .

[31]  P. Whittle.  Risk-sensitive linear/quadratic/gaussian control , 1981, Advances in Applied Probability.

[32]  W. Fleming,et al.  Risk sensitive optimal control and differential games , 1992 .

[33]  Andrew G. Barto,et al.  Learning to Act Using Real-Time Dynamic Programming , 1995, Artif. Intell..

[34]  Robert R. Bitmead,et al.  Adaptive optimal control , 1990 .

[35]  Mayuresh V. Kothare,et al.  Efficient robust constrained model predictive control with a time varying terminal constraint set , 2003, Syst. Control. Lett..

[36]  S. Dreyfus.  Dynamic Programming and the Calculus of Variations , 1960 .

[37]  Tamer Basar,et al.  Dual Control Theory , 2001 .

[38]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[39]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[40]  J. Doyle,et al.  Robust and optimal control , 1995, Proceedings of 35th IEEE Conference on Decision and Control.

[41]  Zhenghong Yu,et al.  Worst-case formulations of model predictive control for systems with bounded parameters , 1997, Autom..

[42]  Manfred Morari,et al.  Robust Model Predictive Control , 1987, 1987 American Control Conference.

[43]  M. Wendt,et al.  Robust model predictive control under chance constraints , 2000 .

[44]  Michael Nikolaou,et al.  Robust stability analysis of constrained l1‐norm model predictive control , 1993 .

[45]  B. L. Cooley,et al.  Optimal feedback control strategies for state-space systems with stochastic parameters , 1998, IEEE Trans. Autom. Control..