Smooth Alternating Direction Methods for Nonsmooth Constrained Convex Optimization

We propose two new alternating direction methods to solve "fully" nonsmooth constrained convex problems. Our algorithms have the best known worst-case iteration-complexity guarantee under mild assumptions for both the objective residual and feasibility gap. Through theoretical analysis, we show how to update all the algorithmic parameters automatically with clear impact on the convergence performance. We also provide a representative numerical example showing the advantages of our methods over the classical alternating direction methods using a well-known feasibility problem.

[1]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[2]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[3]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[4]  Paul Tseng,et al.  Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints , 1991, Math. Oper. Res..

[5]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[6]  Krzysztof C. Kiwiel,et al.  Free-Steering Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints , 1997, Math. Oper. Res..

[7]  Paul Tseng,et al.  Alternating Projection-Proximal Methods for Convex Programming and Variational Inequalities , 1997, SIAM J. Optim..

[8]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[9]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[10]  O. SIAMJ.,et al.  PROX-METHOD WITH RATE OF CONVERGENCE O(1/t) FOR VARIATIONAL INEQUALITIES WITH LIPSCHITZ CONTINUOUS MONOTONE OPERATORS AND SMOOTH CONVEX-CONCAVE SADDLE POINT PROBLEMS∗ , 2004 .

[11]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[12]  Yurii Nesterov,et al.  Excessive Gap Technique in Nonsmooth Convex Minimization , 2005, SIAM J. Optim..

[13]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[14]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[15]  Johan A. K. Suykens,et al.  Application of a Smoothing Technique to Decomposition in Convex Optimization , 2008, IEEE Transactions on Automatic Control.

[16]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[17]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[18]  Junfeng Yang,et al.  Alternating Direction Algorithms for 1-Problems in Compressive Sensing , 2009, SIAM J. Sci. Comput..

[19]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[20]  Truong Q. Nguyen,et al.  An Augmented Lagrangian Method for Total Variation Video Restoration , 2011, IEEE Transactions on Image Processing.

[21]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[22]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[23]  A. Belloni,et al.  Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming , 2011 .

[24]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[25]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[26]  Xiaoming Yuan,et al.  On the O(1/t) Convergence Rate of Alternating Direction Method with Logarithmic-Quadratic Proximal Regularization , 2012, SIAM J. Optim..

[27]  Asuman E. Ozdaglar,et al.  On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[28]  Ion Necoara,et al.  Computational Complexity of Inexact Gradient Augmented Lagrangian Methods: Application to Constrained MPC , 2013, SIAM J. Control. Optim..

[29]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[30]  Marc Teboulle,et al.  A fast dual proximal gradient algorithm for convex minimization and applications , 2014, Oper. Res. Lett..

[31]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[32]  Patrick L. Combettes,et al.  Best Approximation from the Kuhn-Tucker Set of Composite Monotone Inclusions , 2014, 1401.8005.

[33]  V. Cevher,et al.  A Primal-Dual Algorithmic Framework for Constrained Convex Minimization , 2014, 1406.5403.

[34]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[35]  Volkan Cevher,et al.  Constrained convex minimization via model-based excessive gap , 2014, NIPS.

[36]  Arindam Banerjee,et al.  Bregman Alternating Direction Method of Multipliers , 2013, NIPS.

[37]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[38]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[39]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[40]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[41]  慧 廣瀬 A Mathematical Introduction to Compressive Sensing , 2015 .

[42]  Yunmei Chen,et al.  An Accelerated Linearized Alternating Direction Method of Multipliers , 2014, SIAM J. Imaging Sci..

[43]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[44]  Damek Davis,et al.  Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme , 2014, SIAM J. Optim..

[45]  Shiqian Ma,et al.  On the Global Linear Convergence of the ADMM with MultiBlock Variables , 2014, SIAM J. Optim..

[46]  A. Yurtsever,et al.  Universal Primal-Dual Proximal-Gradient Methods , 2015 .

[47]  Shiqian Ma,et al.  Iteration Complexity Analysis of Multi-block ADMM for a Family of Convex Minimization Without Strong Convexity , 2015, Journal of Scientific Computing.

[48]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[49]  Marc Teboulle,et al.  On the rate of convergence of the proximal alternating linearized minimization algorithm for convex problems , 2016, EURO J. Comput. Optim..

[50]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[51]  Xiaoming Yuan,et al.  On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function , 2017, Comput. Optim. Appl..

[52]  Shiqian Ma,et al.  An Extragradient-Based Alternating Direction Method for Convex Minimization , 2017, Found. Comput. Math..