Don't relax: early stopping for convex regularization

We consider the problem of designing efficient regularization algorithms when regularization is encoded by a (strongly) convex functional. Unlike classical penalization methods based on a relaxation approach, we propose an iterative method where regularization is achieved via early stopping. Our results show that the proposed procedure achieves the same recovery accuracy as penalization methods, while naturally integrating computational considerations. An empirical analysis on a number of problems provides promising results with respect to the state of the art.

[1]  Jean-François Aujol,et al.  Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA , 2015, SIAM J. Optim..

[2]  P. L. Combettes,et al.  Dualization of Signal Recovery Problems , 2009, 0907.0436.

[3]  Martin Burger,et al.  Iterative total variation schemes for nonlinear inverse problems , 2009 .

[4]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[5]  Wotao Yin,et al.  An Iterative Regularization Method for Total Variation-Based Image Restoration , 2005, Multiscale Model. Simul..

[6]  Robert Tibshirani,et al.  The Entire Regularization Path for the Support Vector Machine , 2004, J. Mach. Learn. Res..

[7]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[8]  Y. Yao,et al.  On Early Stopping in Gradient Descent Learning , 2007 .

[9]  Gabriele Steidl,et al.  First order algorithms in variational image processing , 2014, ArXiv.

[10]  Xavier Bresson,et al.  Bregmanized Nonlocal Regularization for Deconvolution and Sparse Reconstruction , 2010, SIAM J. Imaging Sci..

[11]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[12]  A. Caponnetto,et al.  Optimal Rates for the Regularized Least-Squares Algorithm , 2007, Found. Comput. Math..

[13]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[14]  R. Boţ,et al.  Iterative regularization with a general penalty term—theory and application to L1 and TV regularization , 2012 .

[15]  Stanley Osher,et al.  A Guide to the TV Zoo , 2013 .

[16]  Léon Bottou,et al.  The Tradeoffs of Large Scale Learning , 2007, NIPS.

[17]  Lorenzo Rosasco,et al.  Learning with Incremental Iterative Regularization , 2014, NIPS.

[18]  Lin He,et al.  Error estimation for Bregman iterations and inverse scale space methods in image restoration , 2007, Computing.

[19]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[20]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[21]  Marc Teboulle,et al.  A fast dual proximal gradient algorithm for convex minimization and applications , 2014, Oper. Res. Lett..

[22]  Martin J. Wainwright,et al.  Early stopping for non-parametric regression: An optimal data-dependent stopping rule , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[23]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[24]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[25]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[26]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[27]  Bin Yu,et al.  Boosting with early stopping: Convergence and consistency , 2005, math/0508276.

[28]  Lorenzo Rosasco,et al.  On regularization algorithms in learning theory , 2007, J. Complex..

[29]  Yin Zhang,et al.  Fixed-Point Continuation for l1-Minimization: Methodology and Convergence , 2008, SIAM J. Optim..

[30]  S. Osher,et al.  Sparse Recovery via Differential Inclusions , 2014, 1406.7728.

[31]  Benjamin Recht,et al.  Sharp Time–Data Tradeoffs for Linear Inverse Problems , 2015, IEEE Transactions on Information Theory.

[32]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[33]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.