OSGA: a fast subgradient algorithm with optimal complexity

This paper presents an algorithm for approximately minimizing a convex function in simple, not necessarily bounded convex, finite-dimensional domains, assuming only that function values and subgradients are available. No global information about the objective function is needed apart from a strong convexity parameter (which can be put to zero if only convexity is known). The worst case number of iterations needed to achieve a given accuracy is independent of the dimension and—apart from a constant factor—best possible under a variety of smoothness assumptions on the objective function.

[1]  Xiangrui Meng,et al.  Accelerating Nesterov's Method for Strongly Convex Functions with Lipschitz Gradient , 2011, 1109.6058.

[2]  Samuel Burer,et al.  A First-Order Smoothing Technique for a Class of Large-Scale Linear Programs , 2014, SIAM J. Optim..

[3]  Guanghui Lan,et al.  Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization , 2013, Mathematical Programming.

[4]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[5]  Masoud Ahookhosh,et al.  Optimal subgradient algorithms with application to large-scale linear inverse problems , 2014, 1402.7291.

[6]  Yurii Nesterov,et al.  Primal-dual subgradient methods for convex problems , 2005, Math. Program..

[7]  Clóvis C. Gonzaga,et al.  An Optimal Algorithm for Constrained Differentiable Convex Optimization , 2013, SIAM J. Optim..

[8]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[9]  Yurii Nesterov,et al.  Rounding of convex sets and efficient gradient methods for linear programming problems , 2004, Optim. Methods Softw..

[10]  Jacek Gondzio,et al.  Matrix-free interior point method for compressed sensing problems , 2012, Mathematical Programming Computation.

[11]  Marc Teboulle,et al.  Interior Gradient and Proximal Methods for Convex and Conic Optimization , 2006, SIAM J. Optim..

[12]  A. Juditsky,et al.  Deterministic and Stochastic Primal-Dual Subgradient Algorithms for Uniformly Convex Minimization , 2014 .

[13]  Necdet Serhat Aybat,et al.  A First-Order Augmented Lagrangian Method for Compressed Sensing , 2010, SIAM J. Optim..

[14]  Yurii Nesterov,et al.  Unconstrained Convex Minimization in Relative Scale , 2003, Math. Oper. Res..

[15]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[16]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[17]  Guanghui Lan,et al.  Primal-dual first-order methods with O (1/e) iteration-complexity for cone programming. , 2011 .

[18]  Peter Richtárik,et al.  Improved Algorithms for Convex Minimization in Relative Scale , 2011, SIAM J. Optim..

[19]  S. V. N. Vishwanathan,et al.  A Quasi-Newton Approach to Nonsmooth Convex Optimization Problems in Machine Learning , 2008, J. Mach. Learn. Res..

[20]  O. Axelsson,et al.  ON THE RATE OF CONVERGENCE OF THE CONJUGATE GRADIENT METHOD FOR LINEAR OPERATORS IN HILBERT SPACE , 2002 .

[21]  Masoud Ahookhosh,et al.  Optimal subgradient algorithms for large-scale convex optimization in simple domains , 2015, Numerical Algorithms.

[22]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[23]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[24]  Y. Nesterov,et al.  Primal-dual subgradient methods for minimizing uniformly convex functions , 2010, 1401.1792.

[25]  Ming Gu,et al.  ParNes: a rapidly convergent algorithm for accurate recovery of sparse and approximately sparse signals , 2009, Numerical Algorithms.

[26]  Renato D. C. Monteiro,et al.  Primal-dual first-order methods with $${\mathcal {O}(1/\epsilon)}$$ iteration-complexity for cone programming , 2011, Math. Program..

[27]  A. Neumaier,et al.  Solving nonsmooth convex optimization with complexity O ( ε − 1 / 2 ) , 2015 .

[28]  Amir Beck,et al.  The CoMirror algorithm for solving nonsmooth constrained convex problems , 2010, Oper. Res. Lett..

[29]  Masoud Ahookhosh,et al.  An optimal subgradient algorithm for large-scale bound-constrained convex optimization , 2015, Math. Methods Oper. Res..

[30]  Emmanuel J. Candès,et al.  Templates for convex cone problems with applications to sparse signal recovery , 2010, Math. Program. Comput..

[31]  Owe Axelsson.  Iterative Solution Methods: The Rate of Convergence of the Conjugate Gradient Method , 1994 .

[32]  O. Axelsson,et al.  On the rate of convergence of the preconditioned conjugate gradient method , 1986 .

[33]  Masoud Ahookhosh,et al.  An Optimal Subgradient Algorithm with Subspace Search for Costly Convex Optimization Problems , 2018, Bulletin of the Iranian Mathematical Society.

[34]  A. Neumaier,et al.  Solving structured nonsmooth convex optimization with complexity $$\mathcal {O}(\varepsilon ^{-1/2})$$O(ε-1/2) , 2017 .

[35]  Clóvis C. Gonzaga,et al.  Fine tuning Nesterov’s steepest descent algorithm for differentiable convex programming , 2012, Mathematical Programming.

[36]  Yurii Nesterov,et al.  Universal gradient methods for convex optimization problems , 2015, Math. Program..

[37]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.