Variable Metric Forward-Backward Algorithm for Composite Minimization Problems

We present a forward-backward-based algorithm to minimize a sum of a differentiable function and a nonsmooth function, both being possibly nonconvex. The main contribution of this work is to consider the challenging case where the nonsmooth function corresponds to a sum of non-convex functions, resulting from composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function. The proposed variable metric Composite Function Forward-Backward algorithm (C2FB) circumvents the explicit, and often challenging, computation of the proximity operator of the composite functions through a majorize-minimize approach. Precisely, each composite function is majorized using a linear approximation of the differentiable function, which allows one to apply the proximity step only to the sum of the nonsmooth functions. We prove the convergence of the algorithm iterates to a critical point of the objective function leveraging the Kurdyka-\L ojasiewicz inequality. The convergence is guaranteed even if the proximity operators are computed inexactly, considering relative errors. We show that the proposed approach is a generalization of reweighting methods, with convergence guarantees. In particular, applied to the log-sum function, our algorithm reduces to a generalized version of the celebrated reweighted $\ell_1$ method. Finally, we show through simulations on an image processing problem that the proposed C2FB algorithm necessitates less iterations to converge and leads to better critical points compared with traditional reweighting methods and classic forward-backward algorithms.

[1]  Tao Luo,et al.  Improving "Fast Iterative Shrinkage-Thresholding Algorithm": Faster, Smarter, and Greedier , 2018, SIAM J. Sci. Comput..

[2]  Dmitriy Drusvyatskiy,et al.  Nonsmooth optimization using Taylor-like models: error bounds, convergence, and termination criteria , 2016, Mathematical Programming.

[3]  Tuomo Valkonen,et al.  Relaxed Gauss-Newton methods with applications to electrical impedance tomography , 2020, SIAM J. Imaging Sci..

[4]  Mohamed-Jalal Fadili,et al.  Non-smooth Non-convex Bregman Minimization: Unification and New Algorithms , 2017, Journal of Optimization Theory and Applications.

[5]  Peter Ochs,et al.  Unifying Abstract Inexact Convergence Theorems and Block Coordinate Variable Metric iPiano , 2016, SIAM J. Optim..

[6]  Michael Möller,et al.  Composite Optimization by Nonconvex Majorization-Minimization , 2018, SIAM J. Imaging Sci..

[7]  Marc Teboulle,et al.  First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems , 2017, SIAM J. Optim..

[8]  Dmitriy Drusvyatskiy,et al.  Error Bounds, Quadratic Growth, and Linear Convergence of Proximal Methods , 2016, Math. Oper. Res..

[9]  Prabhu Babu,et al.  Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning , 2017, IEEE Transactions on Signal Processing.

[10]  Émilie Chouzenoux,et al.  A block coordinate variable metric forward–backward algorithm , 2016, Journal of Global Optimization.

[11]  Federica Porta,et al.  On the convergence of a linesearch based proximal-gradient method for nonconvex optimization , 2016, 1605.03791.

[12]  Edouard Pauwels,et al.  Majorization-Minimization Procedures and Convergence of SQP Methods for Semi-Algebraic and Tame Programs , 2014, Math. Oper. Res..

[13]  Stephen J. Wright,et al.  A proximal method for composite minimization , 2008, Mathematical Programming.

[14]  Juan Peypouquet,et al.  Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates , 2015, J. Optim. Theory Appl..

[15]  Thomas Brox,et al.  On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision , 2015, SIAM J. Imaging Sci..

[16]  Jean-Christophe Pesquet,et al.  Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed ${\ell _1}/{\ell _2}$ Regularization , 2014, IEEE Signal Processing Letters.

[17]  Gabriele Steidl,et al.  First order algorithms in variational image processing , 2014, ArXiv.

[18]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[19]  Émilie Chouzenoux,et al.  Variable Metric Forward–Backward Algorithm for Minimizing the Sum of a Differentiable Function and a Convex Function , 2013, Journal of Optimization Theory and Applications.

[20]  Émilie Chouzenoux,et al.  A preconditioned Forward-Backward approach with application to large-scale nonconvex spectral unmixing problems , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[22]  Tuomo Valkonen,et al.  A primal–dual hybrid gradient method for nonlinear operators with applications to MRI , 2013, 1309.5032.

[23]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[24]  Julien Mairal,et al.  Optimization with First-Order Surrogate Functions , 2013, ICML.

[25]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[26]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[27]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Composite Function Minimization with Applications to Nonconvex Nonlinear Programming , 2011, SIAM J. Optim..

[28]  P. L. Combettes,et al.  Proximity for sums of composite functions , 2010, 1007.3535.

[29]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[30]  J. Bolte,et al.  Characterizations of Lojasiewicz inequalities: Subgradient flows, talweg, convexity , 2009 .

[31]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[32]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[33]  Dirk A. Lorenz,et al.  Iterated Hard Shrinkage for Minimization Problems with Sparsity Constraints , 2008, SIAM J. Sci. Comput..

[34]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[35]  Adrian S. Lewis,et al.  Clarke Subgradients of Stratifiable Functions , 2006, SIAM J. Optim..

[36]  E.J. Candes Compressive Sampling , 2022 .

[37]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[38]  D. Hunter,et al.  A Tutorial on MM Algorithms , 2004 .

[39]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[40]  I. Daubechies,et al.  Factoring wavelet transforms into lifting steps , 1998 .

[41]  S. Mallat A wavelet tour of signal processing , 1998 .

[42]  R. Tyrrell Rockafellar,et al.  Convergence Rates in Forward-Backward Splitting , 1997, SIAM J. Optim..

[43]  P. L. Combettes,et al.  The Convex Feasibility Problem in Image Recovery , 1996 .

[44]  Michael C. Ferris,et al.  A Gauss—Newton method for convex composite optimization , 1995, Math. Program..

[45]  K. Kurdyka,et al.  Wf-stratification of subanalytic functions and the Lojasiewicz inequality , 1994 .

[46]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[47]  Stephen J. Wright Convergence of an inexact algorithm for composite nonsmooth optimization , 1990 .

[48]  James V. Burke,et al.  Descent methods for composite nondifferentiable optimization problems , 1985, Math. Program..

[49]  Ya-Xiang Yuan,et al.  On the superlinear convergence of a trust region algorithm for nonsmooth optimization , 1985, Math. Program..

[50]  M. J. D. Powell,et al.  On the global convergence of trust region algorithms for unconstrained minimization , 1984, Math. Program..

[51]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[52]  R. Fletcher A model algorithm for composite nondifferentiable optimization problems , 1982 .

[53]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[54]  J. Moreau Proximité et dualité dans un espace hilbertien , 1965 .