Variable Smoothing for Weakly Convex Composite Functions

We study minimization of a structured objective function, being the sum of a smooth function and a composition of a weakly convex function with a linear operator. Applications include image reconstruction problems with regularizers that introduce less bias than the standard convex regularizers. We develop a variable smoothing algorithm, based on the Moreau envelope with a decreasing sequence of smoothing parameters, and prove a complexity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}(\epsilon ^{-3})$$\end{document}O(ϵ-3) to achieve an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document}ϵ-approximate solution. This bound interpolates between the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}(\epsilon ^{-2})$$\end{document}O(ϵ-2) bound for the smooth case and the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {O}}(\epsilon ^{-4})$$\end{document}O(ϵ-4) bound for the subgradient method. Our complexity bound is in line with other works that deal with structured nonsmoothness of weakly convex functions.

[1]  Jianqing Fan,et al.  COMMENTS ON « WAVELETS IN STATISTICS : A REVIEW , 2009 .

[2]  Dmitriy Drusvyatskiy,et al.  Stochastic model-based minimization of weakly convex functions , 2018, SIAM J. Optim..

[3]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[4]  Po-Ling Loh,et al.  Statistical consistency and asymptotic normality for high-dimensional robust M-estimators , 2015, ArXiv.

[5]  J. Tukey,et al.  The Fitting of Power Series, Meaning Polynomials, Illustrated on Band-Spectroscopic Data , 1974 .

[6]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[7]  Prateek Jain,et al.  Efficient Algorithms for Smooth Minimax Optimization , 2019, NeurIPS.

[8]  Grace Wahba,et al.  LASSO-Patternsearch algorithm with application to ophthalmology and genomic data. , 2006, Statistics and its interface.

[9]  Volkan Cevher,et al.  A Smooth Primal-Dual Optimization Framework for Nonsmooth Composite Convex Minimization , 2015, SIAM J. Optim..

[10]  Dmitriy Drusvyatskiy,et al.  Stochastic subgradient method converges at the rate O(k-1/4) on weakly convex functions , 2018, ArXiv.

[11]  Jianqing Fan,et al.  Comments on «Wavelets in statistics: A review» by A. Antoniadis , 1997 .

[12]  Ivan W. Selesnick,et al.  Convex Denoising using Non-Convex Tight Frame Regularization , 2015, IEEE Signal Processing Letters.

[13]  F. Giannessi.  Variational Analysis and Generalized Differentiation , 2006 .

[14]  Guoyin Li,et al.  Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods , 2016, Foundations of Computational Mathematics.

[15]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[16]  Ernö Robert Csetnek,et al.  On the convergence rate of a forward-backward type primal-dual splitting algorithm for convex optimization problems , 2015 .

[17]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[18]  Yuantao Gu,et al.  Nonconvex Sparse Logistic Regression With Weakly Convex Regularization , 2017, IEEE Transactions on Signal Processing.

[19]  Saeed Ghadimi,et al.  Accelerated gradient methods for nonconvex nonlinear and stochastic programming , 2013, Mathematical Programming.

[20]  Stephen J. Wright,et al.  Variable Smoothing for Weakly Convex Composite Functions , 2020, Journal of Optimization Theory and Applications.

[21]  Stephen J. Wright,et al.  A proximal method for composite minimization , 2008, Mathematical Programming.

[22]  Radu Ioan Bot,et al.  Convergence Analysis for a Primal-Dual Monotone + Skew Splitting Algorithm with Applications to Total Variation Minimization , 2012, Journal of Mathematical Imaging and Vision.

[23]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[24]  Ilker Bayram,et al.  On the Convergence of the Iterative Shrinkage/Thresholding Algorithm With a Weakly Convex Penalty , 2015, IEEE Transactions on Signal Processing.

[25]  Djemel Ziou,et al.  Image Quality Metrics: PSNR vs. SSIM , 2010, 2010 20th International Conference on Pattern Recognition.

[26]  OsherStanley,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[27]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[28]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[29]  R. Boţ,et al.  A variable smoothing algorithm for solving convex optimization problems , 2012, 1207.3254.

[30]  Antonin Chambolle,et al.  Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications , 2017, SIAM J. Optim..

[31]  Ilker Bayram.  On the Convergence of the Iterative Shrinkage/Thresholding Algorithm With a Weakly Convex Penalty , 2016, IEEE Trans. Signal Process..

[32]  Dmitriy Drusvyatskiy,et al.  Efficiency of minimizing compositions of convex functions and smooth maps , 2016, Math. Program..

[33]  Radu Ioan Boţ,et al.  Variable Smoothing for Convex Optimization Problems Using Stochastic Gradients , 2019, Journal of Scientific Computing.

[34]  Stephen J. Wright,et al.  LASSO-Patternsearch Algorithm with Application to Ophthalmology Data , 2006 .

[35]  Adam M. Oberman,et al.  ON PROXIMAL POINT-TYPE ALGORITHMS FOR WEAKLY CONVEX FUNCTIONS AND THEIR CONNECTION TO THE BACKWARD EULER METHOD , 2018 .