Accelerated Iterative Regularization via Dual Diagonal Descent

We propose and analyze an accelerated iterative dual diagonal descent algorithm for the solution of linear inverse problems with general regularization and data-fit functions. In particular , we develop an inertial approach of which we analyze both convergence and stability. Using tools from inexact proximal calculus, we prove early stopping results with optimal convergence rates for additive data-fit terms as well as more general cases, such as the Kullback-Leibler divergence, for which different type of proximal point approximations hold.

[1]  Hedy Attouch,et al.  The Rate of Convergence of Nesterov's Accelerated Forward-Backward Method is Actually Faster Than 1/k2 , 2015, SIAM J. Optim..

[2]  Alexandre M. Bayen,et al.  Accelerated Mirror Descent in Continuous and Discrete Time , 2015, NIPS.

[3]  H. Attouch,et al.  Asymptotic behavior of nonautonomous monotone and subgradient evolution equations , 2016, 1601.00767.

[4]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[5]  Francis R. Bach,et al.  Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning , 2008, NIPS.

[6]  Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects , 2016, 1602.00232.

[7]  Lea Fleischer,et al.  Regularization of Inverse Problems , 1996 .

[8]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[9]  Martin Burger,et al.  ERROR ESTIMATES FOR GENERAL FIDELITIES , 2011 .

[10]  Juan Peypouquet,et al.  A Dynamical Approach to an Inertial Forward-Backward Algorithm for Convex Minimization , 2014, SIAM J. Optim..

[11]  Hédy Attouch,et al.  Viscosity Solutions of Minimization Problems , 1996, SIAM J. Optim..

[12]  Michael Hintermüller,et al.  Subspace Correction Methods for a Class of Nonsmooth and Nonadditive Convex Variational Problems with Mixed L1/L2 Data-Fidelity in Image Processing , 2013, SIAM J. Imaging Sci..

[13]  Barbara Kaltenbacher,et al.  Convergence rates for an iteratively regularized Newton–Landweber iteration in Banach space , 2013 .

[14]  J. Peypouquet,et al.  SPLITTING FORWARD-BACKWARD PENALTY SCHEME FOR CONSTRAINED VARIATIONAL PROBLEMS , 2014, 1408.0974.

[15]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[16]  Jean-François Aujol,et al.  Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA , 2015, SIAM J. Optim..

[17]  Jean-François Aujol,et al.  The Differential Inclusion Modeling FISTA Algorithm and Optimality of Convergence Rate in the Case b $\leq3$ , 2018, SIAM J. Optim..

[18]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[19]  OsherStanley,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[20]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[21]  Stanley Osher,et al.  A Guide to the TV Zoo , 2013 .

[22]  H. Attouch,et al.  A Dynamical Approach to Convex Minimization Coupling Approximation with the Steepest Descent Method , 1996 .

[23]  C. Zălinescu.  Convex analysis in general vector spaces , 2002 .

[24]  Andreas Neubauer,et al.  On Nesterov acceleration for Landweber iteration of linear ill-posed problems , 2016 .

[25]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[26]  Lorenzo Rosasco,et al.  Iterative Regularization via Dual Diagonal Descent , 2016, Journal of Mathematical Imaging and Vision.

[27]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[28]  B. Lemaire,et al.  Convergence of diagonally stationary sequences in convex optimization , 1994 .

[29]  Claudio Estatico,et al.  Preconditioned iterative regularization in Banach spaces , 2013, Comput. Optim. Appl..

[30]  H. Attouch,et al.  Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[31]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[32]  Lorenzo Rosasco,et al.  Don't relax: early stopping for convex regularization , 2017, ArXiv.

[33]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[34]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[35]  Laetitia Paoli,et al.  Asymptotics for some vibro‐impact problems with a linear dissipation term , 2007 .

[36]  R. Boţ,et al.  Iterative regularization with a general penalty term—theory and application to L1 and TV regularization , 2012 .

[37]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[38]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[39]  Carola-Bibiane Schönlieb,et al.  Infimal Convolution of Data Discrepancies for Mixed Noise Removal , 2016, SIAM J. Imaging Sci..

[40]  Martin Burger,et al.  Modern regularization methods for inverse problems , 2018, Acta Numerica.

[41]  Juan Peypouquet,et al.  Coupling Forward-Backward with Penalty Schemes and Parallel Splitting for Constrained Variational Inequalities , 2011, SIAM J. Optim..

[42]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[43]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[44]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[45]  Mohamed-Jalal Fadili,et al.  Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter Selection , 2014, SIAM J. Imaging Sci..

[46]  A. Bakushinsky,et al.  Iterative Methods for Approximate Solution of Inverse Problems , 2005 .

[47]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[48]  B. Kaltenbacher,et al.  Iterative methods for nonlinear ill-posed problems in Banach spaces: convergence and applications to parameter identification problems , 2009 .

[49]  Barbara Kaltenbacher,et al.  Iterative Regularization Methods for Nonlinear Ill-Posed Problems , 2008, Radon Series on Computational and Applied Mathematics.

[50]  Martin Burger,et al.  Iterative total variation schemes for nonlinear inverse problems , 2009 .

[51]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[52]  H. Attouch,et al.  Combining fast inertial dynamics for convex optimization with Tikhonov regularization , 2016, 1602.01973.

[53]  Jean-François Aujol,et al.  Convergence rate of inertial Forward–Backward algorithm beyond Nesterov’s rule , 2018, Mathematical Programming.

[54]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[55]  H. Attouch,et al.  Asymptotic behavior of coupled dynamical systems with multiscale aspects , 2009, 0904.0397.

[56]  Lin He,et al.  Error estimation for Bregman iterations and inverse scale space methods in image restoration , 2007, Computing.

[57]  Hédy Attouch,et al.  Inertial Forward–Backward Algorithms with Perturbations: Application to Tikhonov Regularization , 2018, J. Optim. Theory Appl..

[58]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[59]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[60]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.