Distributed Optimization Using the Primal-Dual Method of Multipliers

In this paper, we propose the primal-dual method of multipliers (PDMM) for distributed optimization over a graph. In particular, we optimize a sum of convex functions defined over a graph, where every edge in the graph carries a linear equality constraint. In designing the new algorithm, an augmented primal-dual Lagrangian function is constructed which smoothly captures the graph topology. It is shown that a saddle point of the constructed function provides an optimal solution of the original problem. Further under both the synchronous and asynchronous updating schemes, PDMM has the convergence rate of <inline-formula><tex-math notation="LaTeX">$O(1/K)$</tex-math></inline-formula> (where <inline-formula><tex-math notation="LaTeX">$K$</tex-math></inline-formula> denotes the iteration index) for general closed, proper, and convex functions. Other properties of PDMM such as convergence speeds versus different parameter-settings and resilience to transmission failure are also investigated through the experiments of distributed averaging.

[1]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[2]  椹木 義一,et al.  Theory of multiobjective optimization , 1985 .

[3]  Walid Hachem,et al.  Analysis of Sum-Weight-Like Algorithms for Averaging in Wireless Sensor Networks , 2012, IEEE Transactions on Signal Processing.

[4]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[5]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[6]  Martin J. Wainwright,et al.  Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling , 2010, IEEE Transactions on Automatic Control.

[7]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[8]  Arindam Banerjee,et al.  Online Alternating Direction Method , 2012, ICML.

[9]  Richard Heusdens,et al.  A distributed algorithm for robust LCMV beamforming , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Stephen P. Boyd,et al.  Randomized gossip algorithms , 2006, IEEE Transactions on Information Theory.

[11]  Pascal Bianchi,et al.  A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization , 2014, IEEE Transactions on Automatic Control.

[12]  Richard Heusdens,et al.  Convergence of Min-Sum-Min Message-Passing for Quadratic Optimization , 2014, ECML/PKDD.

[13]  Asuman E. Ozdaglar,et al.  On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[14]  Richard Heusdens,et al.  Bi-alternating direction method of multipliers over graphs , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Richard Heusdens Distributed Convex Optimization A Study on the Primal-Dual Method of Multipliers , 2015 .

[16]  Ali H. Sayed,et al.  Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks , 2011, IEEE Transactions on Signal Processing.

[17]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[18]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[19]  Richard Heusdens,et al.  On the convergence rate of the bi-alternating direction method of multipliers , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  D. Sontag 1 Introduction to Dual Decomposition for Inference , 2010 .

[21]  Richard Heusdens,et al.  Large Scale LP Decoding with Low Complexity , 2013, IEEE Communications Letters.

[22]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[23]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message Passing for Quadratic Optimization , 2006, IEEE Transactions on Information Theory.

[24]  Richard Heusdens,et al.  Linear Coordinate-Descent Message Passing for Quadratic Optimization , 2012, Neural Computation.

[25]  Xiangfeng Wang,et al.  Asynchronous Distributed ADMM for Large-Scale Optimization—Part I: Algorithm and Convergence Analysis , 2015, IEEE Transactions on Signal Processing.

[26]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[27]  Ramesh Govindan,et al.  Understanding packet delivery performance in dense wireless sensor networks , 2003, SenSys '03.

[28]  Tommi S. Jaakkola,et al.  Introduction to dual composition for inference , 2011 .

[29]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[30]  Richard Heusdens,et al.  On simplifying the primal-dual method of multipliers , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).