Distributed SDDM Solvers: Theory & Applications

In this paper, we propose distributed solvers for systems of linear equations given by symmetric diagonally dominant M-matrices based on the parallel solver of Spielman and Peng. We propose two versions of the solvers, where in the first, full communication in the network is required, while in the second communication is restricted to the R-Hop neighborhood between nodes for some $R \geq 1$. We rigorously analyze the convergence and convergence rates of our solvers, showing that our methods are capable of outperforming state-of-the-art techniques. Having developed such solvers, we then contribute by proposing an accurate distributed Newton method for network flow optimization. Exploiting the sparsity pattern of the dual Hessian, we propose a Newton method for network flow optimization that is both faster and more accurate than state-of-the-art techniques. Our method utilizes the distributed SDDM solvers for determining the Newton direction up to any arbitrary precision $\epsilon >0$. We analyze the properties of our algorithm and show superlinear convergence within a neighborhood of the optimal. Finally, in a set of experiments conducted on randomly generated and barbell networks, we demonstrate that our approach is capable of significantly outperforming state-of-the-art techniques.

[1]  D. Bertsekas,et al.  Projected Newton methods and optimization of multicommodity flows , 1982, CDC 1982.

[2]  A. Barrett Network Flows and Monotropic Optimization. , 1984 .

[3]  E. F. Kaasschieter,et al.  Preconditioned conjugate gradients for solving singular systems , 1988 .

[4]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[5]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[6]  O. Axelsson Iterative solution methods , 1995 .

[7]  Anil Joshi Topics in optimization and sparse linear systems , 1997 .

[8]  J. Sun,et al.  Applying a Newton Method to Strictly Convex Separable Network Quadratic Programs , 1998, SIAM J. Optim..

[9]  Steven H. Low,et al.  Optimization flow control—I: basic algorithm and convergence , 1999, TNET.

[10]  Steven H. Low,et al.  Optimization flow control with Newton‐like algorithm , 2000, Telecommun. Syst..

[11]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[12]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[13]  Béla Bollobás,et al.  The degree sequence of a scale‐free random graph process , 2001, Random Struct. Algorithms.

[14]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[15]  Bruce Hendrickson,et al.  Support Theory for Preconditioning , 2003, SIAM J. Matrix Anal. Appl..

[16]  Dimitri P. Bertsekas,et al.  Convex Analysis and Optimization , 2003 .

[17]  B. Schölkopf,et al.  A Regularization Framework for Learning from Graph Data , 2004, ICML 2004.

[18]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[19]  O. Sorkine Differential Representations for Mesh Processing , 2006 .

[20]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[21]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Gary L. Miller,et al.  A linear work, O(n1/6) time, parallel algorithm for solving planar Laplacians , 2007, SODA '07.

[23]  Daniel A. Spielman,et al.  Faster approximate lossy generalized flow via interior point algorithms , 2008, STOC.

[24]  Bruce Hendrickson,et al.  Solving Elliptic Finite Element Systems in Near-Linear Time with Support Preconditioners , 2004, SIAM J. Numer. Anal..

[25]  Daniel Cohen-Or,et al.  Dynamic harmonic fields for surface processing , 2009, Comput. Graph..

[26]  Aleksander Madry,et al.  Faster Generation of Random Spanning Trees , 2009, 2009 50th Annual IEEE Symposium on Foundations of Computer Science.

[27]  Asuman E. Ozdaglar,et al.  A distributed newton method for network optimization , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[28]  Asuman E. Ozdaglar,et al.  Approximate Primal Solutions and Rate Analysis for Dual Subgradient Methods , 2008, SIAM J. Optim..

[29]  Allan D. Jepson,et al.  Benchmarking Image Segmentation Algorithms , 2009, International Journal of Computer Vision.

[30]  Gilles Bertrand,et al.  Watershed Cuts: Minimum Spanning Forests and the Drop of Water Principle , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Alejandro Ribeiro Ergodic Stochastic Optimization Algorithms for Wireless Communication and Networking , 2010, IEEE Trans. Signal Process..

[32]  Tim Roughgarden,et al.  Fully Distributed Algorithms for Convex Optimization Problems , 2010, SIAM J. Optim..

[33]  Asuman E. Ozdaglar,et al.  A distributed Newton method for Network Utility Maximization , 2010, 49th IEEE Conference on Decision and Control (CDC).

[34]  Alejandro Ribeiro,et al.  Separation Principles in Wireless Networking , 2010, IEEE Transactions on Information Theory.

[35]  Gary L. Miller,et al.  Approaching optimality for solving SDD systems , 2010, ArXiv.

[36]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Camille Couprie,et al.  Power Watershed: A Unifying Graph-Based Optimization Framework , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Gary L. Miller,et al.  A Nearly-m log n Time Solver for SDD Linear Systems , 2011, 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science.

[39]  Shang-Hua Teng,et al.  Spectral Sparsification of Graphs , 2008, SIAM J. Comput..

[40]  Alejandro Ribeiro,et al.  Accelerated dual descent for network optimization , 2011, Proceedings of the 2011 American Control Conference.

[41]  Near linear-work parallel SDD solvers, low-diameter decomposition, and low-stretch subgraphs , 2011, SPAA '11.

[42]  Shaoshuai Mou,et al.  An asynchronous distributed algorithm for solving a linear algebraic equation , 2013, 52nd IEEE Conference on Decision and Control.

[43]  Aleksander Madry,et al.  Navigating Central Path with Electrical Flows: From Flows to Matchings, and Back , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[44]  Alex Olshevsky,et al.  Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-Agent Control , 2014, 1411.4186.

[45]  Shang-Hua Teng,et al.  Nearly-Linear Time Algorithms for Preconditioning and Solving Symmetric, Diagonally Dominant Linear Systems , 2006, SIAM J. Matrix Anal. Appl..

[46]  Eric Eaton,et al.  On the Degree Distribution of Pólya Urn Graph Processes , 2014, ArXiv.

[47]  Devavrat Shah,et al.  Solving Systems of Linear Equations: Locally and Asynchronously , 2014, ArXiv.

[48]  Richard Peng,et al.  An efficient parallel solver for SDD linear systems , 2013, STOC.

[49]  Gabriel Taubin,et al.  Laplacian Coordinates for Seeded Image Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.