Line search for averaged operator iteration

Many popular first order algorithms for convex optimization, such as forward-backward splitting, Douglas-Rachford splitting, and the alternating direction method of multipliers (ADMM), can be formulated as averaged iteration of a nonexpansive mapping. In this paper we propose a line search for averaged iteration that preserves the theoretical convergence guarantee, while often accelerating practical convergence. We discuss several general cases in which the additional computational cost of the line search is modest compared to the savings obtained.

[1]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[2]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[3]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[4]  M. Benzi.  Preconditioning techniques for large linear systems: a survey , 2002 .

[5]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[6]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[7]  Stephen P. Boyd,et al.  A Primer on Monotone Operator Methods , 2015 .

[8]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[9]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[10]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[11]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[12]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[13]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[14]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[15]  Pontus Giselsson,et al.  Tight global linear convergence rate bounds for Douglas–Rachford splitting , 2015, Journal of Fixed Point Theory and Applications.

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[18]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[19]  Apostol T. Vassilev,et al.  Analysis of the Inexact Uzawa Algorithm for Saddle Point Problems , 1997 .

[20]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[21]  Heinz H. Bauschke,et al.  The Douglas-Rachford Algorithm for Two (Not Necessarily Intersecting) Affine Subspaces , 2015, SIAM J. Optim..

[22]  Jun Zou,et al.  Nonlinear Inexact Uzawa Algorithms for Linear and Nonlinear Saddle-point Problems , 2006, SIAM J. Optim..

[23]  P. L. Combettes,et al.  Compositions and convex combinations of averaged nonexpansive operators , 2014, 1407.5100.

[24]  Stephen P. Boyd,et al.  Metric selection in fast dual forward-backward splitting , 2015, Autom..

[25]  Stephen P. Boyd,et al.  Metric Selection in Douglas-Rachford Splitting and ADMM , 2014 .