A historical overview of iterative methods

The object of this paper is to present a historical overview of the development of iterative methods for the solution of large sparse systems of linear equations. The emphasis is on methods which are applicable to linear systems arising in the numerical solution of partial differential equations. Aspects to be covered including the methods of L.F. Richardson and of Liebmann as well as relaxation methods used by Southwell and others; the SOR method and extensions such as block SOR methods, and p-cyclic matrices; Chebyshev polynomial methods; alternating direction implicit methods; the SSOR method; approximate matrix factorization methods including the strongly implicit method (SIP) and the incomplete Cholesky method (ICC); fast direct methods; conjugate gradient methods; adaptive methods for the automatic determination of iteration parameters; multigrid methods; methods for nonsymmetric systems; and iterative software. Future developments will be discussed with emphasis on the use of vector and parallel processors.

[1]  T. Manteuffel.  Adaptive procedure for estimating parameters for the nonsymmetric Tchebychev iteration , 1978 .

[2]  P. Swarztrauber.  A direct Method for the Discrete Solution of Separable Elliptic Equations , 1974 .

[3]  L. Ehrlich.  An Ad Hoc SOR Method , 1981 .

[4]  S. Eisenstat,et al.  Variational Iterative Methods for Nonsymmetric Systems of Linear Equations , 1983 .

[5]  R. Varga,et al.  Implicit alternating direction methods , 1959 .

[6]  David R. Kincaid,et al.  Algorithm 586: ITPACK 2C: A FORTRAN Package for Solving Large Sparse Linear Systems by Adaptive Accelerated Iterative Methods , 1982, TOMS.

[7]  C. G. Broyden.  Some generalisations of the theory of successive over-relaxation , 1964 .

[8]  J. Reid.  The Use of Conjugate Gradients for Systems of Linear Equations Possessing “Property A” , 1972 .

[9]  C. Lanczos.  Solution of Systems of Linear Equations by Minimized Iterations1 , 1952 .

[10]  Gian-Carlo Rota,et al.  History of Computing in the Twentieth Century , 1980 .

[11]  C. Jacobi.  Ueber eine neue Auflösungsart der bei der Methode der kleinsten Quadrate vorkommenden lineären Gleichungen , 1845 .

[12]  O. Axelsson.  Conjugate gradient type methods for unsymmetric and inconsistent systems of linear equations , 1980 .

[13]  Garrett Birkhoff,et al.  The Numerical Solution of Elliptic Equations , 1987 .

[14]  Elizabeth H. Cuthill,et al.  A Method of Normalized Block Iteration , 1959, JACM.

[15]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[16]  R. Glowinski,et al.  Computing Methods in Applied Sciences and Engineering , 1974 .

[17]  D. Young.  On the accelerated SSOR method for solving large linear systems , 1977 .

[18]  R. P. Kendall,et al.  An Approximate Factorization Procedure for Solving Self-Adjoint Elliptic Difference Equations , 1968 .

[19]  Kang C. Jea,et al.  Generalized conjugate-gradient acceleration of nonsymmetrizable iterative methods , 1980 .

[20]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .

[21]  David J. Evans,et al.  An Iterative Process for Optimizing Symmetric Successive Over-Relaxation , 1963, Comput. J..

[22]  Self-Consistent Solutions of the Plasma Transport Equations in an Axisymmetric Toroidal System , 1981 .

[23]  Olof B. Widlund,et al.  On the rate of convergence of an alternating direction implicit method in a noncommutative case , 1966 .

[24]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[25]  J. Gillis,et al.  Matrix Iterative Analysis , 1961 .

[26]  H. E. Wrigley.  Accelerating the Jacobi Method for Solving Simultaneous Equations by Chebyshev Extrapolation When the Eigenvalues of the Iteration Matrix are Complex , 1963, Computer/law journal.

[27]  J. H. Verner,et al.  On generalizations of the theory of consistent orderings for successive over-relaxation methods , 1968 .

[28]  R. Sweet.  A Generalized Cyclic Reduction Algorithm , 1974 .

[29]  T. Manteuffel.  An incomplete factorization technique for positive definite linear systems , 1980 .

[30]  D. A. Flanders,et al.  Numerical Determination of Fundamental Modes , 1950 .

[31]  R. P. Fedorenko.  A relaxation method for solving elliptic difference equations , 1962 .

[32]  G. Golub,et al.  Use of fast direct methods for the efficient numerical solution of nonseparable elliptic equations. , 1972 .

[33]  L. Ehrlich.  The Block Symmetric Successive Overrelaxation Method , 1964 .

[34]  J. Daniel.  The Conjugate Gradient Method for Linear and Nonlinear Operator Equations , 1967 .

[35]  Kang C. Jea,et al.  On the simplification of generalized conjugate-gradient methods for nonsymmetrizable linear systems , 1983 .

[36]  J. Ortega.  Introduction to Parallel and Vector Solution of Linear Systems , 1988, Frontiers of Computer Science.

[37]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[38]  W. G. Bickley,et al.  Relaxation Methods in Theoretical Physics , 1947 .

[39]  O. Axelsson.  A generalized SSOR method , 1972 .

[40]  David Young,et al.  Alternating Direction Implicit Methods , 1962, Adv. Comput..

[41]  J. J. Douglas.  On the Numerical Integration of $\frac{\partial ^2 u}{\partial x^2 } + \frac{\partial ^2 u}{\partial y^2 } = \frac{\partial u}{\partial t}$ by Implicit Methods , 1955 .

[42]  David M. Young,et al.  Second-degree iterative methods for the solution of large linear systems , 1972 .

[43]  I. Gustafsson.  A class of first order factorization methods , 1978 .

[44]  G. Shortley,et al.  The Numerical Solution of Laplace's Equation , 1938 .

[45]  Wladimir Markoff,et al.  Über Polynome, die in einem gegebenen Intervalle möglichst wenig von Null abweichen , 1916 .

[46]  N. Bakhvalov.  On the convergence of a relaxation method with natural constraints on the elliptic operator , 1966 .

[47]  J. Meijerink,et al.  An iterative solution method for linear systems of which the coefficient matrix is a symmetric -matrix , 1977 .

[48]  William L. Briggs,et al.  A multigrid tutorial , 1987 .

[49]  D. J. Evans,et al.  The Use of Pre-conditioning in Iterative Methods for Solving Linear Equations with Symmetric Positive Definite Matrices , 1968 .

[50]  O. Widlund.  A Lanczos Method for a Class of Nonsymmetric Systems of Linear Equations , 1978 .

[51]  Richard S. Varga,et al.  $p$-cyclic matrices: A generalization of the Young-Frankel successive overrelaxation scheme. , 1959 .

[52]  Richard S. Varga,et al.  Orderings of the successive overrelaxation scheme , 1959 .

[53]  Robert Piessens,et al.  The evaluation and application of some modified moments , 1973 .

[54]  T. Manteuffel.  The Tchebychev iteration for nonsymmetric linear systems , 1977 .

[55]  L. D. Gates,et al.  A Method of Block Iteration , 1956 .

[56]  G. Habetler,et al.  An Alternating-Direction-Implicit Iteration Technique , 1960 .

[57]  George E. Forsythe,et al.  Solving linear algebraic equations can be interesting , 1953 .

[58]  O. Widlund,et al.  Iterative solution of elliptic systems : and applications to the neutron diffusion equations of reactor physics , 1967 .

[59]  H. L. Stone.  ITERATIVE SOLUTION OF IMPLICIT APPROXIMATIONS OF MULTIDIMENSIONAL PARTIAL DIFFERENTIAL EQUATIONS , 1968 .

[60]  D. Young.  Iterative methods for solving partial difference equations of elliptic type , 1954 .

[61]  Louis A. Hageman,et al.  Iterative Solution of Large Linear Systems. , 1971 .

[62]  G. Shortley.  Use of Tschebyscheff‐Polynomial Operators in the Numerical Solution of Boundary‐Value Problems , 1953 .

[63]  G. Habetler,et al.  Symmetric Successive Overrelaxation In Solving Diffusion Difference Equations , 1961 .

[64]  D. Brandt,et al.  Multi-level adaptive solutions to boundary-value problems math comptr , 1977 .

[65]  C. G. Broyden.  Some aspects of consistent ordering , 1968 .