Decentralized Accelerated Gradient Methods With Increasing Penalty Parameters

In this article, we study the communication, and (sub)gradient computation costs in distributed optimization. We present two algorithms based on the framework of the accelerated penalty method with increasing penalty parameters. Our first algorithm is for smooth distributed optimization, and it obtains the near optimal <inline-formula><tex-math notation="LaTeX">$O(\sqrt{\frac{L}{\epsilon (1-\sigma _2(W))}}\log \frac{1}{\epsilon })$</tex-math></inline-formula> communication complexity, and the optimal <inline-formula><tex-math notation="LaTeX">$O(\sqrt{\frac{L}{\epsilon }})$</tex-math></inline-formula> gradient computation complexity for <inline-formula><tex-math notation="LaTeX">$L$</tex-math></inline-formula>-smooth convex problems, where <inline-formula><tex-math notation="LaTeX">$\sigma _2(W)$</tex-math></inline-formula> denotes the second largest singular value of the weight matrix <inline-formula><tex-math notation="LaTeX">$W$</tex-math></inline-formula> associated to the network, and <inline-formula><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> is the target accuracy. When the problem is <inline-formula><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula>-strongly convex, and <inline-formula><tex-math notation="LaTeX">$L$</tex-math></inline-formula>-smooth, our algorithm has the near optimal <inline-formula><tex-math notation="LaTeX">$O(\sqrt{\frac{L}{\mu (1-\sigma _2(W))}}\log ^2\frac{1}{\epsilon })$</tex-math></inline-formula> complexity for communications, and the optimal <inline-formula><tex-math notation="LaTeX">$O(\sqrt{\frac{L}{\mu }}\log \frac{1}{\epsilon })$</tex-math></inline-formula> complexity for gradient computations. Our communication complexities are only worse by a factor of <inline-formula><tex-math notation="LaTeX">$(\log \frac{1}{\epsilon })$</tex-math></inline-formula> than the lower bounds. Our second algorithm is designed for nonsmooth distributed optimization, and it achieves both the optimal <inline-formula><tex-math notation="LaTeX">$O(\frac{1}{\epsilon \sqrt{1-\sigma _2(W)}})$</tex-math></inline-formula> communication complexity, and <inline-formula><tex-math notation="LaTeX">$O(\frac{1}{\epsilon ^2})$</tex-math></inline-formula> subgradient computation complexity, which match the lower bounds for nonsmooth distributed optimization.