Potential-Function Proofs for First-Order Methods

This note discusses proofs for convergence of first-order methods based on simple potential-function arguments. We cover methods like gradient descent (for both smooth and non-smooth settings), mirror descent, and some accelerated variants.

[1]  Zeyuan Allen Zhu,et al.  Linear Coupling: An Ultimate Unification of Gradient and Mirror Descent , 2014, ITCS.

[2]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[3]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[4]  John C. Duchi.  Introductory lectures on stochastic optimization , 2018, IAS/Park City Mathematics Series.

[5]  Alexandre M. Bayen,et al.  Accelerated Mirror Descent in Continuous and Discrete Time , 2015, NIPS.

[6]  Aharon Ben-Tal,et al.  Lectures on modern convex optimization , 1987 .

[7]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[8]  Andre Wibisono,et al.  A variational perspective on accelerated methods in optimization , 2016, Proceedings of the National Academy of Sciences.

[9]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[10]  S. Vavasis,et al.  A single potential governing convergence of conjugate gradient, accelerated gradient and geometric descent , 2017, 1712.09498.

[11]  Francis Bach,et al.  Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions , 2019, COLT.

[12]  Marc Teboulle,et al.  Performance of first-order methods for smooth convex minimization: a novel approach , 2012, Mathematical Programming.

[13]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[14]  Javier Peña.  Convergence of first-order methods via the convex conjugate , 2017, Oper. Res. Lett..

[15]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[16]  Shai Shalev-Shwartz,et al.  Online Learning and Online Convex Optimization , 2012, Found. Trends Mach. Learn..

[17]  Donghwan Kim,et al.  Optimized first-order methods for smooth convex minimization , 2014, Mathematical Programming.

[18]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[19]  Manfred K. Warmuth,et al.  The Minimax Strategy for Gaussian Density Estimation. pp , 2000, COLT.

[20]  E. Takimoto,et al.  The Minimax Strategy for Gaussian Density Estimation , 2000 .

[21]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[22]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .