Fast gradient descent method for convex optimization problems with an oracle that generates a $(\delta,L)$-model of a function in a requested point

In this article we propose a new concept of a $(\delta,L)$-model of a function which generalizes the concept of the $(\delta,L)$-oracle (Devolder-Glineur-Nesterov). Using this concept we describe the gradient descent method and the fast gradient descent method and we show that many popular methods (composite optimization methods, level methods, conjugate gradient methods, proximal methods) are special cases of proposed methods in this article.

[1]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[2]  Adrien B. Taylor,et al.  Exact Worst-Case Performance of First-Order Methods for Composite Convex Optimization , 2015, SIAM J. Optim..

[3]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[4]  P. Dvurechensky,et al.  Stochastic Intermediate Gradient Method for Convex Problems with Inexact Stochastic Oracle , 2014, 1411.2876.

[5]  Ohad Shamir,et al.  Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization , 2011, ICML.

[6]  Thomas S. Huang,et al.  Bregman distance to L1 regularized logistic regression , 2008, 2008 19th International Conference on Pattern Recognition.

[7]  Julien Mairal,et al.  Optimization with First-Order Surrogate Functions , 2013, ICML.

[8]  A. Tyurin.  Mirror version of similar triangles method for constrained optimization problems , 2017, 1705.09809.

[9]  Martin Jaggi,et al.  Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization , 2013, ICML.

[10]  Olivier Devolder,et al.  Exactness, inexactness and stochasticity in first-order methods for large-scale convex optimization , 2013 .

[11]  Y. Nesterov,et al.  First-order methods with inexact oracle: the strongly convex case , 2013 .

[12]  A. Juditsky,et al.  5 First-Order Methods for Nonsmooth Convex Large-Scale Optimization , I : General Purpose Methods , 2010 .

[13]  Mohamed-Jalal Fadili,et al.  Non-smooth Non-convex Bregman Minimization: Unification and New Algorithms , 2017, Journal of Optimization Theory and Applications.

[14]  Y. Nesterov,et al.  Intermediate gradient methods for smooth convex problems with inexact oracle , 2013 .

[15]  Cristobal Guzman,et al.  On lower complexity bounds for large-scale smooth convex optimization , 2013, J. Complex..

[16]  Guanghui Lan,et al.  Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization , 2013, Mathematical Programming.

[17]  Zaïd Harchaoui,et al.  A Universal Catalyst for First-Order Optimization , 2015, NIPS.

[18]  Zaïd Harchaoui,et al.  Conditional gradient algorithms for norm-regularized smooth convex optimization , 2013, Math. Program..

[19]  Alexander Gasnikov,et al.  Stochastic Intermediate Gradient Method for Convex Problems with Stochastic Inexact Oracle , 2016, Journal of Optimization Theory and Applications.

[20]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[21]  Yurii Nesterov,et al.  Universal gradient methods for convex optimization problems , 2015, Math. Program..

[22]  Yurii Nesterov,et al.  Complexity bounds for primal-dual methods minimizing the model of objective function , 2017, Mathematical Programming.