The Variable Metric Forward-Backward Splitting Algorithm Under Mild Differentiability Assumptions

We study the variable metric forward-backward splitting algorithm for convex minimization problems without the standard assumption of the Lipschitz continuity of the gradient. In this setting, we prove that, by requiring only mild assumptions on the smooth part of the objective function and using several types of line search procedures for determining either the gradient descent stepsizes, or the relaxation parameters, one still obtains weak convergence of the iterates and convergence in the objective function values. Moreover, the $o(1/k)$ convergence rate in the function values is obtained if slightly stronger differentiability assumptions are added. We also illustrate several applications including problems that involve Banach spaces and functions of divergence type.

[1]  Heinz H. Bauschke,et al.  Joint minimization with alternating Bregman proximity operators , 2005 .

[2]  P. L. Combettes,et al.  Compositions and convex combinations of averaged nonexpansive operators , 2014, 1407.5100.

[3]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[4]  José Yunier Bello Cruz,et al.  On the convergence of the forward–backward splitting method with linesearches , 2015, Optim. Methods Softw..

[5]  Kristian Kirsch,et al.  Methods Of Modern Mathematical Physics , 2016 .

[6]  P. L. Combettes,et al.  Variable metric quasi-Fejér monotonicity , 2012, 1206.5705.

[7]  S. Bonettini,et al.  New convergence results for the scaled gradient projection method , 2014, 1406.6601.

[8]  Valeria Ruggiero,et al.  On the Convergence of Primal–Dual Hybrid Gradient Algorithms for Total Variation Image Restoration , 2012, Journal of Mathematical Imaging and Vision.

[9]  Heinz H. Bauschke,et al.  The Baillon-Haddad Theorem Revisited , 2009, 0906.0807.

[10]  Émilie Chouzenoux,et al.  Variable Metric Forward–Backward Algorithm for Minimizing the Sum of a Differentiable Function and a Convex Function , 2013, Journal of Optimization Theory and Applications.

[11]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[12]  Michael A. Saunders,et al.  Proximal Newton-Type Methods for Minimizing Composite Functions , 2012, SIAM J. Optim..

[13]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[14]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[15]  Joram Lindenstrauss,et al.  Classical Banach spaces , 1973 .

[16]  Federica Porta,et al.  Variable Metric Inexact Line-Search-Based Methods for Nonsmooth Optimization , 2015, SIAM J. Optim..

[17]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[18]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[19]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[20]  K. Bredies.  A forward–backward splitting algorithm for the minimization of non-smooth convex functionals in Banach space , 2008, 0807.0778.

[21]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[22]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[23]  G. McCormick,et al.  The Gradient Projection Method under Mild Differentiability Conditions , 1972 .

[24]  Pablo A. Lotito,et al.  A Class of Inexact Variable Metric Proximal Point Algorithms , 2008, SIAM J. Optim..

[25]  R. Tyrrell Rockafellar,et al.  Convergence Rates in Forward-Backward Splitting , 1997, SIAM J. Optim..

[26]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[27]  Paul Tseng,et al.  A coordinate gradient descent method for nonsmooth separable minimization , 2008, Math. Program..

[28]  Volkan Cevher,et al.  Composite self-concordant minimization , 2013, J. Mach. Learn. Res..

[29]  E. A. Nurminskii.  Convergence of the gradient projection method , 1973 .

[30]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[31]  Mark S. C. Reed,et al.  Method of Modern Mathematical Physics , 1972 .

[32]  A. Iusem,et al.  Full convergence of the steepest descent method with inexact line searches , 1995 .

[33]  P. L. Combettes,et al.  Quasi-Fejérian Analysis of Some Optimization Algorithms , 2001 .

[34]  Annalisa Barla,et al.  Alternating Proximal Regularized Dictionary Learning , 2014, Neural Computation.

[35]  José Yunier Bello Cruz,et al.  On the convergence of the proximal forward-backward splitting method with linesearches , 2015 .

[36]  Juan Peypouquet,et al.  Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates , 2015, J. Optim. Theory Appl..

[37]  Lorenzo Rosasco,et al.  Modified Fejér sequences and applications , 2015, Comput. Optim. Appl..

[38]  D. Bertsekas,et al.  TWO-METRIC PROJECTION METHODS FOR CONSTRAINED OPTIMIZATION* , 1984 .

[39]  Barbara Kaltenbacher,et al.  Regularization Methods in Banach Spaces , 2012, Radon Series on Computational and Applied Mathematics.

[40]  E. Polak,et al.  Constrained Minimization Problems in Finite-Dimensional Spaces , 1966 .

[41]  A. Iusem.  On the convergence properties of the projected gradient method for convex optimization , 2003 .

[42]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[43]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[44]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[45]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[46]  Zongben Xu,et al.  Characteristic inequalities of uniformly convex and uniformly smooth Banach spaces , 1991 .