Inexact proximal $\epsilon$-subgradient methods for composite convex optimization problems

We present two approximate versions of the proximal subgradient method for minimizing the sum of two convex functions (not necessarily differentiable). The algorithms involve, at each iteration, inexact evaluations of the proximal operator and approximate subgradients of the functions (namely: the $\epsilon$-subgradients). The methods use different error criteria for approximating the proximal operators. We provide an analysis of the convergence and rate of convergence properties of these methods, considering various stepsize rules, including both, diminishing and constant stepsizes. For the case where one of the functions is smooth, we propose an inexact accelerated version of the proximal gradient method, and prove that the optimal convergence rate for the function values can be achieved.

[1]  G. Minty.  Monotone (nonlinear) operators in Hilbert space , 1962 .

[2]  Lucas E. A. Simões,et al.  ϵ-subgradient algorithms for bilevel convex optimization , 2017, 1703.02648.

[3]  Radu Ioan Bot,et al.  An inertial forward-backward-forward primal-dual splitting algorithm for solving monotone inclusion problems , 2014, Numerical Algorithms.

[4]  José Yunier Bello Cruz,et al.  On proximal subgradient splitting method for minimizing the sum of two nonsmooth convex functions , 2014, 1410.5477.

[5]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[6]  Michel Théra,et al.  An Additive Subfamily of Enlargements of a Maximally Monotone Operator , 2015 .

[7]  Naum Zuselevich Shor,et al.  Minimization Methods for Non-Differentiable Functions , 1985, Springer Series in Computational Mathematics.

[8]  M. Solodov,et al.  A UNIFIED FRAMEWORK FOR SOME INEXACT PROXIMAL POINT ALGORITHMS , 2001 .

[9]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[10]  Alfredo N. Iusem,et al.  On the projected subgradient method for nonsmooth convex optimization in a Hilbert space , 1998, Math. Program..

[11]  Yu. M. Ermol’ev.  On the method of generalized stochastic gradients and quasi-Féjer sequences , 1969 .

[12]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[13]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[14]  R. Díaz Millán,et al.  Two algorithms for solving systems of inclusion problems , 2016, Numerical Algorithms.

[15]  Dimitri P. Bertsekas,et al.  Incremental Subgradient Methods for Nondifferentiable Optimization , 2001, SIAM J. Optim..

[16]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[17]  Dimitri P. Bertsekas,et al.  Convex Optimization Theory , 2009 .

[18]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[19]  Marc Teboulle,et al.  A simplified view of first order methods for optimization , 2018, Math. Program..

[20]  Andrea Simonetto,et al.  Primal Recovery from Consensus-Based Dual Decomposition for Distributed Convex Optimization , 2015, J. Optim. Theory Appl..

[21]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[22]  J. M. Martínez,et al.  Inexact spectral projected gradient methods on convex sets , 2003 .

[23]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[24]  R. Rockafellar,et al.  On the subdifferentiability of convex functions , 1965 .

[25]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[26]  Jingwei Liang,et al.  Local linear convergence analysis of Primal–Dual splitting methods , 2017, 1705.01926.

[27]  Renato D. C. Monteiro,et al.  An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods , 2013, SIAM J. Optim..

[28]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[29]  Guoyin Li,et al.  Global Convergence of Splitting Methods for Nonconvex Composite Optimization , 2014, SIAM J. Optim..

[30]  Dimitri P. Bertsekas,et al.  Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey , 2015, ArXiv.

[31]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[32]  M. Solodov,et al.  A Hybrid Approximate Extragradient – Proximal Point Algorithm Using the Enlargement of a Maximal Monotone Operator , 1999 .

[33]  R. Díaz Millán,et al.  A Direct Splitting Method for Nonsmooth Variational Inequalities , 2014, J. Optim. Theory Appl..

[34]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.