A survey of some aspects of parallel and distributed iterative algorithms

We consider iterative algorithms of the form x := f(z), executed by a parallel or distributed computing system. We first consider synchronous executions of such iterations and study their communication requirements, as well as issues related to processor synchronization. We also discuss the parallelization of iterations of the Gauss-Seidel type. We then consider asynchronous implementations whereby each processor iterates on a different component of x, at its own pace, using the most recently received (but possibly outdated) information on the remaining components of x. While certain algorithms may fail to converge when implemented asynchronously, a large number of positive convergence results is available. We classify asynchronous algorithms into three main categories, depending on the amount of asynchronism they can tolerate, and survey the corresponding convergence results. We also discuss issues related to their termination.

[1]  John N. Tsitsiklis,et al.  Convergence rate and termination of asynchronous iterative algorithms , 1989, ICS '89.

[2]  Dimitri P. Bertsekas,et al.  Distributed Asynchronous Relaxation Methods for Linear Network Flow Problems , 1987 .

[3]  H. Kushner,et al.  Asymptotic properties of distributed and communication stochastic approximation algorithms , 1987 .

[4]  Dimitri P. Bertsekas,et al.  Distributed asynchronous computation of fixed points , 1983, Math. Program..

[5]  Wei Kang Tsai,et al.  Optimal quasi-static routing for virtual circuit networks subjected to stochastic inputs , 1986 .

[6]  Eli Gafni,et al.  Concurrency in heavily loaded neighborhood-constrained systems , 1989, ICDCS.

[7]  P. Spitéri,et al.  Asynchronous relaxation algorithms for optimal control problems , 1986 .

[8]  Tamer Basar,et al.  Asymptotic agreement and convergence of asynchronous stochastic algorithms , 1986, 1986 25th IEEE Conference on Decision and Control.

[9]  Isi Mitrani,et al.  Analysis and Optimum Performance of Two Message-Passing Parallel Processors Synchronized by Rollback , 1984, Perform. Evaluation.

[10]  Debasis Mitra,et al.  A chaotic asynchronous algorithm for computing the fixed point of a nonnegative matrix of unit spectral radius , 1986, JACM.

[11]  J. White,et al.  Reducing the parallel solution time of sparse circuit matrices using reordered Gaussian elimination and relaxation , 1988, 1988., IEEE International Symposium on Circuits and Systems.

[12]  Yousef Saad,et al.  Data Communication in Hypercubes , 1989, J. Parallel Distributed Comput..

[13]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[14]  Oliver A. McBryan,et al.  Hypercube Algorithms and Implementations , 1985, PPSC.

[15]  Gérard M. Baudet,et al.  Asynchronous Iterative Methods for Multiprocessors , 1978, JACM.

[16]  H. T. Kung,et al.  Synchronized and asynchronous parallel algorithms for multiprocessors , 1976 .

[17]  John N. Tritsiklis.  A comparison of Jacobi and Gauss-Seidel parallel iterations , 1989 .

[18]  Michel Dubois,et al.  Sufficient conditions for the convergence of asynchronous iterations , 1989, Parallel Computing.

[19]  Dimitri P. Bertsekas,et al.  The auction algorithm for the minimum cost network flow problem , 1989 .

[20]  John N. Tsitsiklis,et al.  On the stability of asynchronous iterative processes , 1986, 1986 25th IEEE Conference on Decision and Control.

[21]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[22]  J. Donnelly.  Periodic chaotic relaxation , 1971 .

[23]  Sartaj Sahni,et al.  An optimal routing algorithm for mesh-connected Parallel computers , 1980, JACM.

[24]  Decision Systems.,et al.  Communication Aspects of Parallel Processing , 1987 .

[25]  Debasis Mitra,et al.  Asynchronous relaxations for the numerical solution of differential equations by parallel processors , 1985, PPSC.

[26]  Michel Dubois,et al.  Generalized Asynchronous Iterations , 1986, CONPAR.

[27]  Dimitri P. Bertsekas,et al.  Parallel synchronous and asynchronous implementations of the auction algorithm , 1991, Parallel Comput..

[28]  Adam W. Bojanczyk,et al.  Optimal Asynchronous Newton Method for the Solution of Nonlinear Equations , 1984, JACM.

[29]  Stephen S. Lavenberg,et al.  Performance Analysis of a Rollback Method for Distributed Simulation , 1983, International Symposium on Computer Modeling, Measurement and Evaluation.

[30]  F. Robert.  Contraction en norme vectorielle: Convergence d'iterations chaotiques pour des equations non linéaires de point fixe à plusieurs variables , 1976 .

[31]  V. Barbosa.  Concurrency in systems with neighborhood constraints (distributed systems, parallel processing, dining philosophers, simulated annealing) , 1986 .

[32]  J. C. Miellou,et al.  Algorithmes de relaxation chaotique à retards , 1975 .

[33]  D. Bertsekas,et al.  Distributed asynchronous relaxation methods for convex network flow problems , 1987 .

[34]  Edsger W. Dijkstra,et al.  Termination Detection for Diffusing Computations , 1980, Inf. Process. Lett..

[35]  Paul Tseng,et al.  Distributed Computation for Linear Programming Problems Satisfying a Certain Diagonal Dominance Condition , 1990, Math. Oper. Res..

[36]  Dimitri P. Bertsekas,et al.  Dual coordinate step methods for linear network flow problems , 1988, Math. Program..

[37]  H. Kushner,et al.  Stochastic approximation algorithms for parallel and distributed processing , 1987 .

[38]  Michel Dubois,et al.  Performance of Synchronized Iterative Processes in Multiprocessor Systems , 1982, IEEE Transactions on Software Engineering.

[39]  S. Lennart Johnsson,et al.  Optimum Broadcasting and Personalized Communication in Hypercubes , 1989, IEEE Trans. Computers.

[40]  G. C. Fox,et al.  Solving Problems on Concurrent Processors , 1988 .

[41]  Dimitri P. Bertsekas,et al.  Data Networks , 1986 .

[42]  J. Walrand,et al.  Distributed Dynamic Programming , 2022 .

[43]  J. Miellou,et al.  Un critère de convergence pour des méthodes générales de point fixe , 1985 .

[44]  M. Tarazi.  Some convergence results for asynchronous algorithms , 1982 .

[45]  Michel Dubois,et al.  Parallel asynchronous algorithms for discrete data , 1990, JACM.

[46]  D. Bertsekas,et al.  Partially asynchronous, parallel algorithms for network flow and other problems , 1990 .

[47]  William D. Tajibnapis,et al.  A correctness proof of a topology information maintenance protocol for a distributed computer network , 1977, CACM.

[48]  David R. Jefferson,et al.  Virtual time , 1985, ICPP.

[49]  W. Tsai.  Convergence of gradient projection routing methods in an asynchronous stochastic quasi-static virtual circuit network , 1989 .

[50]  P. Spiteri,et al.  Parallel asynchronous algorithms for solving boundary value problems , 1986 .

[51]  F. Robert.  Iterations chaotiques série-parallele pour des équations non lineaires de point fixe , 1974 .

[52]  J. Ortega,et al.  Solution of Partial Differential Equations on Vector and Parallel Computers , 1987 .

[53]  John Tsitsiklis,et al.  Some issues in distributed asynchronous routing in virtual circuit data networks , 1986, 1986 25th IEEE Conference on Decision and Control.

[54]  Baruch Awerbuch,et al.  Complexity of network synchronization , 1985, JACM.

[55]  John N. Tsitsiklis,et al.  Optimal Communication Algorithms for Hypercubes , 1991, J. Parallel Distributed Comput..

[56]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[57]  Leslie Lamport,et al.  Distributed snapshots: determining global states of distributed systems , 1985, TOCS.