Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

We consider the problem of analyzing and designing gradient-based discrete-time optimization algorithms for a class of unconstrained optimization problems having strongly convex objective functions with Lipschitz continuous gradient. By formulating the problem as a robustness analysis problem and making use of a suitable adaptation of the theory of integral quadratic constraints, we establish a framework that allows to analyze convergence rates and robustness properties of existing algorithms and enables the design of novel robust optimization algorithms with prespecified guarantees capable of exploiting additional structure in the objective function.

[1]  Frank Allgöwer,et al.  Sampling strategies for data-driven inference of passivity properties , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[2]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[3]  Adrien B. Taylor,et al.  Performance estimation toolbox (PESTO): Automated worst-case analysis of first-order optimization methods , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[4]  Johan Efberg,et al.  YALMIP : A toolbox for modeling and optimization in MATLAB , 2004 .

[5]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[6]  P. Seiler,et al.  Noncausal FIR Zames-Falb Multiplier Search for Exponential Convergence Rate , 2019, 1902.09473.

[7]  Benjamin Recht,et al.  Exponential convergence bounds using integral quadratic constraints , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[8]  Simon Michalowsky,et al.  The multidimensional n-th order heavy ball method and its application to extremum seeking , 2014, 53rd IEEE Conference on Decision and Control.

[9]  Bikash Joshi,et al.  An Explicit Convergence Rate for Nesterov's Method from SDP , 2018, 2018 IEEE International Symposium on Information Theory (ISIT).

[10]  Eugenius Kaszkurewicz,et al.  Control Perspectives on Numerical Algorithms And Matrix Problems (Advances in Design and Control) (Advances in Design and Control 10) , 2006 .

[11]  Manfred Morari,et al.  Design of First-Order Optimization Algorithms via Sum-of-Squares Programming , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[12]  Christian Feller,et al.  Relaxed Logarithmic Barrier Function Based Model Predictive Control of Linear Systems , 2015, IEEE Transactions on Automatic Control.

[13]  Asuman E. Ozdaglar,et al.  Robust Accelerated Gradient Methods for Smooth Strongly Convex Functions , 2018, SIAM J. Optim..

[14]  C. Ebenbauer,et al.  On a Class of Smooth Optimization Algorithms with Applications in Control , 2012 .

[15]  Michael G. Safonov,et al.  All stability multipliers for repeated MIMO nonlinearities , 2005, Syst. Control. Lett..

[16]  Randy A. Freeman,et al.  Noncausal Zames-Falb Multipliers for Tighter Estimates of Exponential Convergence Rates , 2018, 2018 Annual American Control Conference (ACC).

[17]  Mihailo R. Jovanovic,et al.  Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[18]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[19]  Alejandro Ribeiro,et al.  Analysis of Optimization Algorithms via Integral Quadratic Constraints: Nonstrongly Convex Problems , 2017, SIAM J. Optim..

[20]  Alexandre Megretski,et al.  New results for analysis of systems with repeated nonlinearities , 2001, Autom..

[21]  Fernando Paganini,et al.  Linear matrix inequality methods for robust H 2 analysis: a survey with comparisons , 1999 .

[22]  Chung-Yao Kao,et al.  On Stability of Discrete-Time LTI Systems With Varying Time Delays , 2012, IEEE Transactions on Automatic Control.

[23]  C. Scherer,et al.  Absolute stability analysis of discrete time feedback interconnections , 2017 .

[24]  Michael I. Jordan,et al.  A Lyapunov Analysis of Momentum Methods in Optimization , 2016, ArXiv.

[25]  Johan Löfberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004 .

[26]  Randy A. Freeman,et al.  The Fastest Known Globally Convergent First-Order Method for Minimizing Strongly Convex Functions , 2018, IEEE Control Systems Letters.

[27]  Simon Michalowsky,et al.  Extremum control of linear systems based on output feedback , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[28]  J. Willems,et al.  Some new rearrangement inequalities having application in stability analysis , 1968 .

[29]  Peter Seiler,et al.  Direct Synthesis of Iterative Algorithms With Bounds on Achievable Worst-Case Convergence Rate , 2019, 2020 American Control Conference (ACC).

[30]  Marc Teboulle,et al.  Performance of first-order methods for smooth convex minimization: a novel approach , 2012, Mathematical Programming.

[31]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[32]  P. Falb,et al.  Stability Conditions for Systems with Monotone and Slope-Restricted Nonlinearities , 1968 .

[33]  Adrien B. Taylor,et al.  Efficient first-order methods for convex minimization: a constructive approach , 2018, Mathematical Programming.

[34]  Carsten W. Scherer,et al.  Robust stability and performance analysis based on integral quadratic constraints , 2016, Eur. J. Control.

[35]  Paul Tseng Control perspectives on numerical algorithms and matrix problems , 2008, Math. Comput..

[36]  A. Antipin,et al.  MINIMIZATION OF CONVEX FUNCTIONS ON CONVEX SETS BY MEANS OF DIFFERENTIAL EQUATIONS , 2003 .

[37]  A. Rantzer,et al.  System analysis via integral quadratic constraints , 1997, IEEE Trans. Autom. Control..

[38]  Jan C. Willems,et al.  The Analysis of Feedback Systems , 1971 .

[39]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[40]  Benjamin Recht,et al.  Exponential Stability Analysis via Integral Quadratic Constraints , 2017, ArXiv.