Solving Systems of Linear Equations by Distributed Convex Optimization in the Presence of Stochastic Uncertainty

Abstract In this paper, we propose distributed optimization methods to solve systems of linear equations. We provide convergence analysis for both continuous and discrete time computation models based on linear systems theory. It is shown that the proposed computation approaches work for very general linear equations, scalable with data sets and can be implemented in distributed or parallel fashion. Furthermore, we show that the discrete time algorithm admits constant update step size in the presence of additive uncertainties. This robustness feature makes the approach computationally efficient and supplementary to the existing approaches to deal with uncertainties such as stochastic (sub-)gradient methods and sample averaging.

[1]  L. Richardson.  The Approximate Arithmetical Solution by Finite Differences of Physical Problems Involving Differential Equations, with an Application to the Stresses in a Masonry Dam , 1911 .

[2]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[3]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[4]  Tamio Shimizu,et al.  A Stochastic Approximation Method for Optimization Problems , 1969, Journal of the ACM.

[5]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[6]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[7]  Achiya Dax,et al.  The Convergence of Linear Stationary Iterative Processes for Solving Singular Unstructured Systems of Linear Equations , 1990, SIAM Rev..

[8]  A. Shapiro.  Monte Carlo Sampling Methods , 2003 .

[9]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[10]  Paul H. Siegel,et al.  Gaussian belief propagation solver for systems of linear equations , 2008, 2008 IEEE International Symposium on Information Theory.

[11]  Benjamin Van Roy,et al.  Convergence of Min-Sum Message Passing for Quadratic Optimization , 2006, IEEE Transactions on Information Theory.

[12]  Jing Wang,et al.  Control approach to distributed optimization , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[13]  Jing Wang,et al.  A control perspective for centralized and distributed convex optimization , 2011, IEEE Conference on Decision and Control and European Control Conference.

[14]  Jing Wang,et al.  Distributed Averaging Algorithms Resilient to Communication Noise and Dropouts , 2013, IEEE Transactions on Signal Processing.