Combining Lagrangian decomposition and excessive gap smoothing technique for solving large-scale separable convex optimization problems

A new algorithm for solving large-scale convex optimization problems with a separable objective function is proposed. The basic idea is to combine three techniques: Lagrangian dual decomposition, excessive gap and smoothing. The main advantage of this algorithm is that it automatically and simultaneously updates the smoothness parameters which significantly improves its performance. The convergence of the algorithm is proved under weak conditions imposed on the original problem. The rate of convergence is $O(\frac {1}{k})$, where k is the iteration counter. In the second part of the paper, the proposed algorithm is coupled with a dual scheme to construct a switching variant in a dual decomposition framework. We discuss implementation issues and make a theoretical comparison. Numerical examples confirm the theoretical results.

[1]  N. Komodakis,et al.  MRF Energy Minimization & Beyond via Dual Decomposition , 2010 .

[2]  Philippe Mahey,et al.  Accelerating convergence of a Separable Augmented Lagrangian Algorithm , 2007 .

[3]  Hans-Jakob Lüthi,et al.  Convex risk measures for portfolio optimization and concepts of flexibility , 2005, Math. Program..

[4]  Dimitri P. Bertsekas,et al.  Incremental proximal methods for large scale convex optimization , 2011, Math. Program..

[5]  Y. Nesterov A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[6]  Jonathan E. Spingarn,et al.  Applications of the method of partial inverses to convex programming: Decomposition , 1985, Math. Program..

[7]  M. Moonen,et al.  Improved Dual Decomposition Based Optimization for DSL Dynamic Spectrum Management , 2010, IEEE Transactions on Signal Processing.

[8]  Nikos Komodakis,et al.  MRF Energy Minimization and Beyond via Dual Decomposition , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[10]  Andrzej Ruszczynski,et al.  On Convergence of an Augmented Lagrangian Decomposition Method for Sparse Convex Optimization , 1995, Math. Oper. Res..

[11]  Aswin N. Venkat Distributed Model Predictive Control: Theory and Applications , 2006 .

[12]  Jochen Könemann,et al.  Faster and simpler algorithms for multicommodity flow and other fractional packing problems , 1998, Proceedings 39th Annual Symposium on Foundations of Computer Science (Cat. No.98CB36280).

[13]  Mikael Johansson,et al.  Distributed non-smooth resource allocation over a network , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[14]  Bingsheng He,et al.  Alternating directions based contraction method for generally separable linearly constrained convex programming problems , 2009 .

[15]  Stephen P. Boyd,et al.  Distributed estimation via dual decomposition , 2007, 2007 European Control Conference (ECC).

[16]  D. Bienstock,et al.  Approximating fractional packings and coverings in O(1/ε) iterations , 2006, STOC 2006.

[17]  Shiqian Ma,et al.  Fast Multiple-Splitting Algorithms for Convex Optimization , 2009, SIAM J. Optim..

[18]  Stephen J. Wright,et al.  Distributed MPC Strategies With Application to Power System Automatic Generation Control , 2008, IEEE Transactions on Control Systems Technology.

[19]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[20]  Marc Teboulle,et al.  A proximal-based decomposition method for convex minimization problems , 1994, Math. Program..

[21]  K. Holmberg,et al.  Experiments with primal - dual decomposition and subgradient methods for the uncapacitatied facility location problem , 2001 .

[22]  G. Cohen Optimization by decomposition and coordination: A unified approach , 1978 .

[23]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[24]  Alexandre d'Aspremont,et al.  First-Order Methods for Sparse Covariance Selection , 2006, SIAM J. Matrix Anal. Appl..

[25]  Y. Nesterov A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2) , 1983 .

[26]  B. He,et al.  Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities , 2000 .

[27]  Daniel Bienstock,et al.  Approximating Fractional Packings and Coverings in O(1/epsilon) Iterations , 2006, SIAM J. Comput..

[28]  Nimrod Megiddo,et al.  Horizontal and vertical decomposition in interior point methods for linear programs , 1994 .

[29]  Shih-Ping Han A parallel algorithm for a class of convex programs , 1988 .

[30]  Antonio J. Conejo,et al.  Decomposition Techniques in Mathematical Programming: Engineering and Science Applications , 2006 .

[31]  Martin J. Wainwright,et al.  Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling , 2010, IEEE Transactions on Automatic Control.

[32]  Sanjay Mehrotra,et al.  On the Implementation of a Primal-Dual Interior Point Method , 1992, SIAM J. Optim..

[33]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[34]  Vânia Lúcia Dos Santos Eleutério Finding approximate solutions for large scale linear programs , 2009 .

[35]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[36]  Jonathan Eckstein Parallel alternating direction multiplier decomposition of convex programs , 1994 .

[37]  John S. Baras,et al.  An Optimal Distributed Routing Algorithm using Dual Decomposition Techniques , 2008, Commun. Inf. Syst..

[38]  Kaj Holmberg,et al.  Mean value cross decomposition for nonlinear convex problems , 2006, Optim. Methods Softw..

[39]  MoonenMarc,et al.  Improved dual decomposition based optimization for DSL dynamic spectrum management , 2010 .

[40]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[41]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[42]  Masao Fukushima,et al.  A parallel descent algorithm for convex programming , 1996, Comput. Optim. Appl..

[43]  Robert F. Love,et al.  A Dual Decomposition Method for Minimizing Transportation Costs in Multifacility Location Problems , 1973 .

[44]  Abdelouahed Hamdi,et al.  Decomposition for structured convex programs with smooth multiplier methods , 2005, Appl. Math. Comput..

[45]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[46]  Xiaoming Yuan,et al.  An alternating direction-based contraction method for linearly constrained separable convex programming problems , 2013 .

[47]  Gongyun Zhao,et al.  A Lagrangian Dual Method with Self-Concordant Barriers for Multi-Stage Stochastic Convex Programming , 2005, Math. Program..

[48]  Yurii Nesterov,et al.  Excessive Gap Technique in Nonsmooth Convex Minimization , 2005, SIAM J. Optim..

[49]  Daniela Pucci de Farias,et al.  Decentralized Resource Allocation in Dynamic Networks of Agents , 2008, SIAM J. Optim..

[50]  Johan A. K. Suykens,et al.  Application of a Smoothing Technique to Decomposition in Convex Optimization , 2008, IEEE Transactions on Automatic Control.

[51]  Paschalis Tsiaflakis,et al.  Distributed Spectrum Management Algorithms for Multiuser DSL Networks , 2008, IEEE Transactions on Signal Processing.

[52]  Abdelouahed Hamdi,et al.  Two-level primal-dual proximal decomposition technique to solve large scale optimization problems , 2005, Appl. Math. Comput..

[53]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[54]  Paul Tseng,et al.  Alternating Projection-Proximal Methods for Convex Programming and Variational Inequalities , 1997, SIAM J. Optim..

[55]  Sanjay Mehrotra,et al.  Decomposition Based Interior Point Methods for Two-Stage Stochastic Convex Quadratic Programs with Recourse , 2009, Oper. Res..

[56]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[57]  Jochen Könemann,et al.  Faster and Simpler Algorithms for Multicommodity Flow and Other Fractional Packing Problems , 2007, SIAM J. Comput..

[58]  S. Kontogiorgis,et al.  Alternating direction splitting for block Angular parallel optimization , 1996 .

[59]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[60]  Dinh Quoc Tran,et al.  An Inexact Perturbed Path-Following Method for Lagrangian Decomposition in Large-Scale Separable Convex Optimization , 2011, SIAM J. Optim..

[61]  Hans Joachim Ferreau,et al.  An online active set strategy to overcome the limitations of explicit MPC , 2008 .