Control interpretations for first-order optimization methods

First-order iterative optimization methods play a fundamental role in large scale optimization and machine learning. This paper presents control interpretations for such optimization methods. First, we give loop-shaping interpretations for several existing optimization methods and show that they are composed of basic control elements such as PID and lag compensators. Next, we apply the small gain theorem to draw a connection between the convergence rate analysis of optimization methods and the input-output gain computations of certain complementary sensitivity functions. These connections suggest that standard classical control synthesis tools may be brought to bear on the design of optimization algorithms.

[1]  Shinji Hara,et al.  Properties of sensitivity and complementary sensitivity functions in single-input single-output digital control systems , 1988 .

[2]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[3]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[4]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[5]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[6]  Yun Li,et al.  PID control system analysis, design, and technology , 2005, IEEE Transactions on Control Systems Technology.

[7]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[8]  G. Zames.  On the input-output stability of time-varying nonlinear feedback systems Part one: Conditions derived using concepts of loop gain, conicity, and positivity , 1966 .

[9]  Alexander J. Smola,et al.  A scalable modular convex solver for regularized risk minimization , 2007, KDD '07.

[10]  P. Olver.  Nonlinear Systems , 2013 .

[11]  Jing Wang,et al.  A control perspective for centralized and distributed convex optimization , 2011, IEEE Conference on Decision and Control and European Control Conference.

[12]  Magda Osman,et al.  Control Systems Engineering , 2010 .

[13]  Jing Wang,et al.  Control approach to distributed optimization , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[14]  Benjamin Recht,et al.  Exponential convergence bounds using integral quadratic constraints , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[15]  Shinji Hara,et al.  Properties of complementary sensitivity function in SISO digital control systems , 1989 .

[16]  C. Mohtadi.  Bode's integral theorem for discrete-time systems , 1990 .

[17]  Rick H. Middleton,et al.  Trade-offs in linear control system design , 1991, Autom..

[18]  José M. F. Moura,et al.  Linear Convergence Rate of a Class of Distributed Augmented Lagrangian Algorithms , 2013, IEEE Transactions on Automatic Control.

[19]  Gene F. Franklin,et al.  Feedback Control of Dynamic Systems , 1986 .

[20]  Sonia Martínez,et al.  Distributed convex optimization via continuous-time coordination algorithms with discrete-time communication , 2014, Autom..

[21]  P. N. Paraskevopoulos,et al.  Modern Control Engineering , 2001 .

[22]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[23]  Andre Wibisono,et al.  A variational perspective on accelerated methods in optimization , 2016, Proceedings of the National Academy of Sciences.

[24]  Peter Seiler,et al.  Exponential Decay Rate Conditions for Uncertain Linear Systems Using Integral Quadratic Constraints , 2016, IEEE Transactions on Automatic Control.

[25]  Mi-Ching Tsai,et al.  Robust and Optimal Control , 2014 .