On Convergence Analysis of Gradient Based Primal-Dual Method of Multipliers

Recently, the primal-dual method of multipliers (PDMM) has been proposed and successfully applied to solve a number of decomposable convex optimizations distributedly and iteratively. In this work, we study the gradient based PDMM (GPDMM), where the objective functions are approximated using the gradient information per iteration. It is shown that for a certain class of decomposable convex optimizations, synchronous GPDMM has a sublinear convergence rate of $\mathcal{O}(1/K)$ (where K denotes the iteration index). Experiments on a problem of distributed ridge regularized logistic regression demonstrate the efficiency of synchronous GPDMM.

[1]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[2]  Martin J. Wainwright,et al.  Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling , 2010, IEEE Transactions on Automatic Control.

[3]  Qing Ling,et al.  DLM: Decentralized Linearized Alternating Direction Method of Multipliers , 2015, IEEE Transactions on Signal Processing.

[4]  Richard Heusdens,et al.  Derivation and Analysis of the Primal-Dual Method of Multipliers Based on Monotone Operator Theory , 2017, IEEE Transactions on Signal and Information Processing over Networks.

[5]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[6]  Richard Heusdens,et al.  On Relationship between Primal-Dual Method of Multipliers and Kalman Filter , 2017, ArXiv.

[7]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[8]  Ali H. Sayed,et al.  Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks , 2011, IEEE Transactions on Signal Processing.

[9]  I. M. Otivation.  Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems , 2018 .

[10]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[11]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[12]  Gesualdo Scutari,et al.  NEXT: In-Network Nonconvex Optimization , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[13]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[14]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message Passing for Quadratic Optimization , 2006, IEEE Transactions on Information Theory.

[15]  Jacob Benesty,et al.  Ad hoc microphone array beamforming using the primal-dual method of multipliers , 2016, 2016 24th European Signal Processing Conference (EUSIPCO).

[16]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[17]  Richard Heusdens,et al.  On simplifying the primal-dual method of multipliers , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Richard Heusdens.  Distributed Convex Optimization A Study on the Primal-Dual Method of Multipliers , 2015 .

[19]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[20]  Xiangfeng Wang,et al.  Multi-Agent Distributed Optimization via Inexact Consensus ADMM , 2014, IEEE Transactions on Signal Processing.

[21]  Mingyi Hong,et al.  Prox-PDA: The Proximal Primal-Dual Algorithm for Fast Distributed Nonconvex Optimization and Learning Over Networks , 2017, ICML.

[22]  Thushara D. Abhayapala,et al.  Distributed sparse MVDR beamforming using the bi-alternating direction method of multipliers , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Richard Heusdens,et al.  Distributed Optimization Using the Primal-Dual Method of Multipliers , 2017, IEEE Transactions on Signal and Information Processing over Networks.

[24]  Thushara D. Abhayapala,et al.  Distributed TV-L1 image fusion using PDMM , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[25]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[26]  Jacob Benesty,et al.  Distributed max-SINR speech enhancement with ad hoc microphone arrays , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  Mingyi Hong,et al.  Nonconvex alternating direction method of multipliers for distributed sparse principal component analysis , 2015, 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[28]  Dong Yu,et al.  Automatic Speech Recognition: A Deep Learning Approach , 2014 .

[29]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[30]  W. Bastiaan Kleijn,et al.  Function Splitting and Quadratic Approximation of the Primal-Dual Method of Multipliers for Distributed Optimization Over Graphs , 2018, IEEE Transactions on Signal and Information Processing over Networks.

[31]  J. Pesquet,et al.  A Class of Randomized Primal-Dual Algorithms for Distributed Optimization , 2014, 1406.6404.