Accelerated and Inexact Forward-Backward Algorithms

We propose a convergence analysis of accelerated forward-backward splitting methods for composite function minimization, when the proximity operator is not available in closed form, and can only be computed up to a certain precision. We prove that the $1/k^2$ convergence rate for the function values can be achieved if the admissible errors are of a certain type and satisfy a sufficiently fast decay condition. Our analysis is based on the machinery of estimate sequences first introduced by Nesterov for the study of accelerated gradient descent algorithms. Furthermore, we give a global complexity analysis, taking into account the cost of computing admissible approximations of the proximal point. An experimental analysis is also presented.

[1]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[2]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[3]  Pablo Tamayo,et al.  Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[4]  Benar Fux Svaiter,et al.  An Inexact Hybrid Generalized Proximal Point Algorithm and Some New Results on the Theory of Bregman Functions , 2000, Math. Oper. Res..

[5]  M. Solodov,et al.  A Hybrid Approximate Extragradient – Proximal Point Algorithm Using the Enlargement of a Maximal Monotone Operator , 1999 .

[6]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[7]  J. Moreau.  Fonctions convexes duales et points proximaux dans un espace hilbertien , 1962 .

[8]  Massimo Fornasier,et al.  Theoretical Foundations and Numerical Methods for Sparse Recovery , 2010, Radon Series on Computational and Applied Mathematics.

[9]  Francis R. Bach,et al.  Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning , 2008, NIPS.

[10]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[11]  Osman Güer.  On the convergence of the proximal point algorithm for convex minimization , 1991 .

[12]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[13]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[14]  C. Zălinescu.  Convex analysis in general vector spaces , 2002 .

[15]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[16]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[17]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[18]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[19]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[20]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[21]  Renato D. C. Monteiro,et al.  An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods , 2013, SIAM J. Optim..

[22]  Y. Nesterov.  A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2) , 1983 .

[23]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[24]  P. L. Combettes,et al.  Dualization of Signal Recovery Problems , 2009, 0907.0436.

[25]  Benar Fux Svaiter,et al.  Error bounds for proximal point subproblems and associated inexact proximal point algorithms , 2000, Math. Program..

[26]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[27]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[28]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[29]  Lorenzo Rosasco,et al.  Solving Structured Sparsity Regularization with Proximal Methods , 2010, ECML/PKDD.

[30]  Heinz H. Bauschke,et al.  The Baillon-Haddad Theorem Revisited , 2009, 0906.0807.

[31]  A. Auslender.  Numerical methods for nondifferentiable convex optimization , 1987 .

[32]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[33]  F. Bach,et al.  Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning) , 2011 .

[34]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[35]  Regina Sandra Burachik,et al.  A Relative Error Tolerance for a Family of Generalized Proximal Point Methods , 2001, Math. Oper. Res..

[36]  M. Solodov,et al.  A Comparison of Rates of Convergence of Two Inexact Proximal Point Algorithms , 2000 .

[37]  Mohamed-Jalal Fadili,et al.  Group sparsity with overlapping partition functions , 2011, 2011 19th European Signal Processing Conference.

[38]  Valeria Ruggiero,et al.  On the Convergence of Primal–Dual Hybrid Gradient Algorithms for Total Variation Image Restoration , 2012, Journal of Mathematical Imaging and Vision.

[39]  Claude Lemaréchal,et al.  Convergence of some algorithms for convex minimization , 1993, Math. Program..

[40]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[41]  M. Solodov,et al.  A UNIFIED FRAMEWORK FOR SOME INEXACT PROXIMAL POINT ALGORITHMS , 2001 .

[42]  R. Monteiro,et al.  Convergence rate of inexact proximal point methods with relative error criteria for convex optimization , 2010 .

[43]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[44]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[45]  A. Iusem,et al.  Enlargement of Monotone Operators with Applications to Variational Inequalities , 1997 .

[46]  Antonin Chambolle,et al.  A l1-Unified Variational Framework for Image Restoration , 2004, ECCV.

[47]  Lorenzo Rosasco,et al.  A Primal-Dual Algorithm for Group Sparse Regularization with Overlapping Groups , 2010, NIPS.

[48]  Yoram Singer,et al.  Efficient Online and Batch Learning Using Forward Backward Splitting , 2009, J. Mach. Learn. Res..

[49]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[50]  Francis R. Bach,et al.  Structured Variable Selection with Sparsity-Inducing Norms , 2009, J. Mach. Learn. Res..

[51]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[52]  Lixin Shen,et al.  Efficient First Order Methods for Linear Composite Regularizers , 2011, ArXiv.

[53]  Nelly Pustelnik,et al.  Nested Iterative Algorithms for Convex Constrained Image Recovery Problems , 2008, SIAM J. Imaging Sci..

[54]  Paul Tseng,et al.  Approximation accuracy, gradient methods, and error bound for structured convex optimization , 2010, Math. Program..

[55]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[56]  Jonathan Eckstein,et al.  Approximate iterations in Bregman-function-based proximal algorithms , 1998, Math. Program..

[57]  Alexander J. Zaslavski,et al.  Convergence of a Proximal Point Method in the Presence of Computational Errors in Hilbert Spaces , 2010, SIAM J. Optim..

[58]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[59]  G. Teschke,et al.  Tikhonov replacement functionals for iteratively solving nonlinear operator equations , 2005 .

[60]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[61]  B. Martinet,et al.  R'egularisation d''in'equations variationnelles par approximations successives , 1970 .

[62]  Van,et al.  A gene-expression signature as a predictor of survival in breast cancer. , 2002, The New England journal of medicine.

[63]  Lorenzo Rosasco,et al.  A Regularization Approach to Nonlinear Variable Selection , 2010, AISTATS.

[64]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[65]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[66]  Otmar Scherzer,et al.  Variational Methods in Imaging , 2008, Applied mathematical sciences.

[67]  Yudong D. He,et al.  A Gene-Expression Signature as a Predictor of Survival in Breast Cancer , 2002 .

[68]  I. Daubechies,et al.  Iteratively solving linear inverse problems under general convex constraints , 2007 .

[69]  Bertolt Eicke.  Iteration methods for convexly constrained ill-posed problems in hilbert space , 1992 .

[70]  Ashutosh Sabharwal,et al.  Convexly constrained linear inverse problems: iterative least-squares and regularization , 1998, IEEE Trans. Signal Process..

[71]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[72]  Jean-Philippe Vert,et al.  Group lasso with overlap and graph lasso , 2009, ICML '09.

[73]  B. Lemaire.  About the Convergence of the Proximal Method , 1992 .

[74]  Bingsheng He,et al.  An Accelerated Inexact Proximal Point Algorithm for Convex Minimization , 2012, J. Optim. Theory Appl..

[75]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[76]  P. Zhao,et al.  Grouped and Hierarchical Model Selection through Composite Absolute Penalties , 2007 .

[77]  K. Bredies.  A forward–backward splitting algorithm for the minimization of non-smooth convex functionals in Banach space , 2008, 0807.0778.

[78]  A. Iusem,et al.  A proximal point method for nonsmooth convex optimization problems in Banach spaces , 1997 .

[79]  Naseer Shahzad,et al.  Strong convergence of a proximal point algorithm with general errors , 2012, Optim. Lett..

[80]  P. Zhao,et al.  The composite absolute penalties family for grouped and hierarchical variable selection , 2009, 0909.0411.

[81]  R. Cominetti.  Coupling the Proximal Point Algorithm with Approximation Methods , 1997 .

[82]  Gaël Varoquaux,et al.  Total Variation Regularization for fMRI-Based Prediction of Behavior , 2011, IEEE Transactions on Medical Imaging.