An overview on the eigenvalue computation for matrices

This paper sketches the research developments in the area of computational methods for solving the eigenvalue problems and how the methods developed relate to each other as they evolve over centuries. This is an attempt to write a complete overview on the research on computational aspects of eigenvalue problem, emphasize the history of methods that still play a role and some of those that no longer are considered to be on the main track but are somehow related to the present techniques in some smaller steps. This contribution brings out the state-of-the-art of the algorithms for solving large-scale eigenvalue problems for both symmetric and nonsymmetric matrices separately, thereby clearly drawing a comparison between the differences in the algorithms in practical use for the two. Some of the methods or modifications to the earlier methods that have been notable at the turn of the 21st century will also be covered through this paper under "Modem Approaches". Also as the standard eigenvalue problem has become better understood, in a numerical sense, progress has been made in the generalized eigenvalue problem and this will briefly cover developments in solving these problems too.

[1]  Gerard L. G. Sleijpen,et al.  Jacobi-Davidson Style QR and QZ Algorithms for the Reduction of Matrix Pencils , 1998, SIAM J. Sci. Comput..

[2]  Ilse C. F. Ipsen Computing an Eigenvector with Inverse Iteration , 1997, SIAM Rev..

[3]  G. Stewart On the Sensitivity of the Eigenvalue Problem $Ax = \lambda Bx$ , 1972 .

[4]  J. H. Wilkinson,et al.  Handbook for Automatic Computation. Vol II, Linear Algebra , 1973 .

[5]  Kesheng Wu,et al.  Thick-Restart Lanczos Method for Symmetric Eigenvalue Problems , 1998, IRREGULAR.

[6]  M. Géradin,et al.  On the practical use of the lanczos algorithm in finite element applications to vibration and bifurcation problems , 1983 .

[7]  W. Feller,et al.  New matrix transformations for obtaining characteristic vectors , 1951 .

[8]  A. C. Aitken XX.—Studies in Practical Mathematics. II. The Evaluation of the Latent Roots and Latent Vectors of a Matrix , 1938 .

[9]  P. Henrici Bounds for iterates, inverses, spectral variation and fields of values of non-normal matrices , 1962 .

[10]  V. Hari On sharp quadratic convergence bounds for the serial Jacobi methods , 1991 .

[11]  P. J. Eberlein Errata: A Jacobi-Like Method for the Automatic Computation of Eigenvalues and Eigenvectors , 1962 .

[12]  David S. Watkins,et al.  POLYNOMIAL EIGENVALUE PROBLEMS WITH HAMILTONIAN STRUCTURE , 2002 .

[13]  Haesun Park,et al.  Self-scaling fast rotations for stiff and equality-constrained linear least squares problems , 1996 .

[14]  J. Cullum,et al.  A Practical Procedure for Computing Eigenvalues of Large Sparse Nonsymmetric Matrices , 1986 .

[15]  Gene H. Golub,et al.  Matrix computations , 1983 .

[16]  Andreas Acrivos,et al.  The instability of the steady flow past spheres and disks , 1993, Journal of Fluid Mechanics.

[17]  S. Petiton Parallel subspace method for non-Hermitian eigenproblems on the Connection Machine (CM2) , 1992 .

[18]  Y. Saad,et al.  Complex shift and invert strategies for real matrices , 1987 .

[19]  Gerard L. G. Sleijpen,et al.  A Jacobi-Davidson Iteration Method for Linear Eigenvalue Problems , 1996, SIAM J. Matrix Anal. Appl..

[20]  Friedrich L. Bauer,et al.  On certain methods for expanding the characteristic polynomial , 1959, Numerische Mathematik.

[21]  Jane Cullum,et al.  A Lanczos procedure for the modal analysis of very large nonsymmetric matrices , 1984, The 23rd IEEE Conference on Decision and Control.

[22]  D. Sorensen,et al.  A Truncated RQ Iteration for Large Scale Eigenvalue Calculations , 1998 .

[23]  A. C. Aitken XXV.—On Bernoulli's Numerical Solution of Algebraic Equations , 1927 .

[24]  E. Ovtchinnikov,et al.  Successive eigenvalue relaxation: a new method for the generalized eigenvalue problem and convergence estimates , 2001, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[25]  Bo Kågström,et al.  An Algorithm for Numerical Computation of the Jordan Normal Form of a Complex Matrix , 1980, TOMS.

[26]  H. Hotelling Analysis of a complex of statistical variables into principal components. , 1933 .

[27]  C. Jordan,et al.  Mémoire sur les formes bilinéaires. , 1874 .

[28]  David S. Watkins,et al.  The transmission of shifts and shift blurring in the QR algorithm , 1996 .

[29]  D. Calvetti,et al.  AN IMPLICITLY RESTARTED LANCZOS METHOD FOR LARGE SYMMETRIC EIGENVALUE PROBLEMS , 1994 .

[30]  A. Ostrowski On the convergence of the Rayleigh quotient iteration for the computation of the characteristic roots and vectors. I , 1957 .

[31]  D. Young,et al.  A Survey of Numerical Mathematics , 1988 .

[32]  Heinz Rutishauser Anwendungen des Quotienten-Differenzen-Algorithmus , 1954 .

[33]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[34]  J. Neumann,et al.  Numerical inverting of matrices of high order , 1947 .

[35]  R. Natarajan An Arnoldi-based iterative scheme for nonsymmetric matrix pencils arising in finite element stability problems , 1992 .

[36]  J. B. Rosser,et al.  Separation of close eigenvalues af a real symmetric matrix , 1951 .

[37]  P. Henrici On the Speed of Convergence of Cyclic and Quasicyclic Jacobi Methods for Computing Eigenvalues of Hermitian Matrices , 1958 .

[38]  J. Demmel Computing stable eigendecompositions of matrices , 1986 .

[39]  Robert C. Ward,et al.  Balancing the Generalized Eigenvalue Problem , 1981 .

[40]  D. Sorensen Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[41]  Axel Ruhe Computing nonlinear eigenvalues with spectral transformation Arnoldi , 1996 .

[42]  Nicholas J. Higham,et al.  Structured Backward Error and Condition of Generalized Eigenvalue Problems , 1999, SIAM J. Matrix Anal. Appl..

[43]  W. Gentleman Error analysis of QR decompositions by Givens transformations , 1975 .

[44]  David D. Morrison Remarks on the Unitary Triangularization of a Nonsymmetric Matrix , 1960, JACM.

[45]  S. Kaniel Estimates for Some Computational Techniques - in Linear Algebra , 1966 .

[46]  Zhishun A. Liu,et al.  A Look Ahead Lanczos Algorithm for Unsymmetric Matrices , 1985 .

[47]  B. Parlett,et al.  Accurate singular values and differential qd algorithms , 1994 .

[48]  Christopher C. Paige,et al.  The computation of eigenvalues and eigenvectors of very large sparse matrices , 1971 .

[49]  A. Turing ROUNDING-OFF ERRORS IN MATRIX PROCESSES , 1948 .

[50]  Y. Saad On the Rates of Convergence of the Lanczos and the Block-Lanczos Methods , 1980 .

[51]  On Kublanovskaya''s approach to the solution of the generalized latent value problem for functional , 1983 .

[52]  B. Parlett,et al.  The Lanczos algorithm with selective orthogonalization , 1979 .

[53]  G. Stewart,et al.  An Algorithm for Generalized Matrix Eigenvalue Problems. , 1973 .

[54]  David S. Watkins Performance of the QZ Algorithm in the Presence of Infinite Eigenvalues , 2000, SIAM J. Matrix Anal. Appl..

[55]  J. Greenstadt A method for finding roots of arbitrary matrices , 1955 .

[56]  Alston S. Householder,et al.  Unitary Triangularization of a Nonsymmetric Matrix , 1958, JACM.

[57]  J. Hadamard,et al.  Essai sur l'étude des fonctions données par leur développement de Taylor , 1892 .

[58]  F. L. Bauer Sequential Reduction to Tridiagonal Form , 1959 .

[60]  Friedrich L. Bauer On Modern Matrix Iteration Processes of Bernoulli and Graeffe Type , 1958, JACM.

[61]  Roland W. Freund,et al.  An Implementation of the Look-Ahead Lanczos Algorithm for Non-Hermitian Matrices , 1993, SIAM J. Sci. Comput..

[62]  Ed Anderson,et al.  LAPACK Users' Guide , 1995 .

[63]  A. Householder A Class of Methods for Inverting Matrices , 1958 .

[64]  G. Stewart Matrix Algorithms, Volume II: Eigensystems , 2001 .

[65]  J. Cullum,et al.  Lanczos algorithms for large symmetric eigenvalue computations , 1985 .

[66]  C. Paige Computational variants of the Lanczos method for the eigenproblem , 1972 .

[67]  Arnold Schönhage,et al.  Zur Konvergenz des Jacobi-Verfahrens , 1961 .

[68]  Roland W. Freund,et al.  QMRPACK: a package of QMR algorithms , 1996, TOMS.

[69]  Bo Kågström,et al.  Algorithm 560: JNF, An Algorithm for Numerical Computation of the Jordan Normal Form of a Complex Matrix [F2] , 1980, TOMS.

[70]  C. Jacobi,et al.  C. G. J. Jacobi's Gesammelte Werke: Über ein leichtes Verfahren, die in der Theorie der Sacularstorungen vorkommenden Gleichungen numerisch aufzulosen , 1846 .

[71]  Y. Saad,et al.  Numerical solution of large Lyapunov equations , 1989 .

[72]  J. G. F. Francis,et al.  The QR Transformation - Part 2 , 1962, Comput. J..

[73]  Rudolf Zurmühl,et al.  Matrizen und ihre Anwendungen 1 , 1997 .

[74]  H. Rutishauser The Jacobi method for real symmetric matrices , 1966 .

[75]  R. Ward The Combination Shift $QZ$ Algorithm , 1975 .

[76]  J. H. Wilkinson Global convergene of tridiagonal QR algorithm with origin shifts , 1968 .

[77]  R. R. Whitehead,et al.  A numerical approach to nuclear shell-model calculations , 1972 .

[78]  Ivan Slapničar,et al.  Accurate Symmetric Eigenreduction by a Jacobi Method , 1993 .

[79]  G. Forsythe Singularity and near Singularity in Numerical Analysis , 1958 .

[80]  P. Dooren The Computation of Kronecker's Canonical Form of a Singular Pencil , 1979 .

[81]  Beresford N. Parlett,et al.  Canonical decomposition of Hessenberg matrices , 1967 .

[82]  Amy Nicole Langville,et al.  Google's PageRank and beyond - the science of search engine rankings , 2006 .

[83]  F. R. Gantmakher The Theory of Matrices , 1984 .

[84]  R. A. Brooker,et al.  The method of Lanczos for calculating the characteristic roots and vectors of a real symmetric matrix , 1956 .

[85]  E. Davidson The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices , 1975 .

[86]  Beresford N. Parlett,et al.  The Software Scene in the Extraction of Eigenvalues from Sparse Matrices , 1984 .

[87]  J. Reid,et al.  Tracking the Progress of the Lanczos Algorithm for Large Symmetric Eigenproblems , 1981 .

[88]  Robert L. Causey,et al.  Computing Eigenvalues of Non-Hermitian Matrices by Methods of Jacobi Type , 1958 .

[89]  J. G. F. Francis,et al.  The QR Transformation A Unitary Analogue to the LR Transformation - Part 1 , 1961, Comput. J..

[90]  James Hardy Wilkinson,et al.  Householder's Method for the Solution of the Algebraic Eigenproblem , 1960, Comput. J..

[91]  James M. Ortega,et al.  An error analysis of Householder's method for the symmetric eigenvalue problem , 1962 .

[92]  B. Parlett Analysis of Algorithms for Reflections in Bisectors , 1971 .

[93]  Gene H. Golub,et al.  The Lanczos-Arnoldi algorithm and controllability , 1984 .

[94]  D. Calvetti,et al.  Iterative methods for the computation of a few eigenvalues of a large symmetric matrix , 1996 .

[95]  V. Hari,et al.  A note on a one-sided Jacobi algorithm , 1989 .

[96]  James Hardy Wilkinson,et al.  Kronecker''s canonical form and the QZ algorithm , 1979 .

[97]  David A. Pope,et al.  Maximizing Functions of Rotations—Experiments Concerning Speed of Diagonalization of Symmetric Matrices Using Jacobi's Method , 1957, JACM.

[98]  C. Paige Error Analysis of the Lanczos Algorithm for Tridiagonalizing a Symmetric Matrix , 1976 .

[99]  Yunkai Zhou,et al.  Studies on Jacobi–Davidson, Rayleigh quotient iteration, inverse iteration generalized Davidson and Newton updates , 2006, Numer. Linear Algebra Appl..

[100]  Chao Yang,et al.  Convergence analysis of an inexact truncated RQ-iteration. , 1998 .

[101]  Steve Batterson,et al.  Rayleigh quotient iteration for nonsymmetric matrices , 1990 .

[102]  James Demmel,et al.  On a Block Implementation of Hessenberg Multishift QR Iteration , 1989, Int. J. High Speed Comput..

[103]  Jack Dongarra,et al.  Templates for the Solution of Algebraic Eigenvalue Problems , 2000, Software, environments, tools.

[104]  Gene H. Golub,et al.  The block Lanczos method for computing eigenvalues , 2007, Milestones in Matrix Computation.

[105]  Volker Mehrmann,et al.  Structure-Preserving Methods for Computing Eigenpairs of Large Sparse Skew-Hamiltonian/Hamiltonian Pencils , 2001, SIAM J. Sci. Comput..

[106]  Axel Ruhe Perturbation bounds for means of eigenvalues and invariant subspaces , 1970 .

[107]  C. Reinsch,et al.  Balancing a matrix for calculation of eigenvalues and eigenvectors , 1969 .

[108]  V. Mehrmann The Autonomous Linear Quadratic Control Problem: Theory and Numerical Solution , 1991 .

[109]  B. Parlett The Symmetric Eigenvalue Problem , 1981 .

[110]  Y. Saad Variations on Arnoldi's method for computing eigenelements of large unsymmetric matrices , 1980 .

[111]  Jagdish J. Modi,et al.  Parallel algorithms and matrix computation , 1988 .

[112]  P. Horst A method of factor analysis by means of which all coordinates of the factor matrix are given simultaneously , 1937 .

[113]  H. Rutishauser Der Quotienten-Differenzen-Algorithmus , 1954 .

[114]  R. Mises,et al.  Praktische Verfahren der Gleichungsauflösung . , 1929 .

[115]  J. H. Wilkinson Note on the quadratic convergence of the cyclic Jacobi process , 1962 .

[116]  Martin H. Gutknecht,et al.  A Completed Theory of the Unsymmetric Lanczos Process and Related Algorithms, Part I , 1992, SIAM J. Matrix Anal. Appl..

[117]  V. Kublanovskaya On some algorithms for the solution of the complete eigenvalue problem , 1962 .

[118]  Ming Gu Finding Well-Conditioned Similarities to Block-Diagonalize Nonsymmetric Matrices Is NP-Hard , 1995, J. Complex..

[119]  D. Sorensen,et al.  4. The Implicitly Restarted Arnoldi Method , 1998 .

[120]  Youcef Saad,et al.  Overview of Krylov subspace methods with applications to control problems , 1989 .

[121]  James Demmel,et al.  Jacobi's Method is More Accurate than QR , 1989, SIAM J. Matrix Anal. Appl..

[122]  J. H. Wilkinson The algebraic eigenvalue problem , 1966 .

[123]  K. Singhal,et al.  On roots of functional lambda matrices , 1983 .

[124]  Kesheng Wu,et al.  Thick-Restart Lanczos Method for Large Symmetric Eigenvalue Problems , 2000, SIAM J. Matrix Anal. Appl..

[125]  F. L. Bauer Das Verfahren der Treppeniteration und verwandte Verfahren zur Lösung algebraischer Eigenwertprobleme , 1957 .

[126]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[127]  J. H. Wilkinson,et al.  Handbook for Automatic Computation: Linear Algebra (Grundlehren Der Mathematischen Wissenschaften, Vol 186) , 1986 .

[128]  Peter Benner,et al.  The Symplectic Eigenvalue Problem, the Butterfly Form, the SR Algorithm, and the Lanczos Method , 1998 .

[129]  Bo Kågström,et al.  Extracting partial canonical structure for large scale eigenvalue problems , 2004, Numerical Algorithms.

[130]  Zhang Zhenyue,et al.  A new shift of the QL algorithm for irreducible symmetric tridiagonal matrices , 1985 .

[131]  C. Donald LaBudde,et al.  The reduction of an arbitrary real square matrix to tridiagonal form using similarity transformations , 1963 .

[132]  P. Mantegazza,et al.  Efficient solution of quadratic eigenproblems arising in dynamic analysis of structures , 1977 .

[133]  Axel Ruhe The Rational Krylov Algorithm for Nonlinear Matrix Eigenvalue Problems , 2003 .

[134]  Jack J. Dongarra,et al.  Matrix Eigensystem Routines — EISPACK Guide Extension , 1977, Lecture Notes in Computer Science.

[135]  Peter Benner,et al.  Numerical Computation of Deflating Subspaces of Skew-Hamiltonian/Hamiltonian Pencils , 2002, SIAM J. Matrix Anal. Appl..

[136]  H. W. Turnbull,et al.  An Introduction to the Theory of Canonical Matrices , 1932, Nature.

[137]  M. Gutknecht A Completed Theory of the Unsymmetric Lanczos Process and Related Algorithms. Part II , 1994, SIAM J. Matrix Anal. Appl..

[138]  J. Reid,et al.  On the Automatic Scaling of Matrices for Gaussian Elimination , 1972 .

[139]  James Hardy Wilkinson,et al.  Convergence of the LR, QR, and Related Algorithms , 1965, Comput. J..

[140]  Gene H. Golub,et al.  Ill-conditioned eigensystems and the computation of the Jordan canonical form , 1975, Milestones in Matrix Computation.

[141]  Danny C. Sorensen,et al.  Implicit Application of Polynomial Filters in a k-Step Arnoldi Method , 1992, SIAM J. Matrix Anal. Appl..

[142]  O. Perron Zur Theorie der Matrices , 1907 .

[143]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[144]  H. V. D. Vorst,et al.  Jacobi-Davidson style QR and QZ algorithms for the partial reduction of matrix pencils , 1996 .

[145]  J. Smillie,et al.  The dynamics of Rayleigh quotient iteration , 1989 .

[146]  Axel Ruhe,et al.  Rational Krylov for Large Nonlinear Eigenproblems , 2004, PARA.

[147]  J. Olsen,et al.  Passing the one-billion limit in full configuration-interaction (FCI) calculations , 1990 .

[148]  G. Golub,et al.  150 years old and still alive: eigenproblems , 1997 .

[149]  J. H. Wilkinson Error analysis of floating-point computation , 1960 .

[150]  G. Stewart Introduction to matrix computations , 1973 .

[151]  F. J. Corbató,et al.  On the Coding of Jacobi's Method for Computing Eigenvalues and Eigenvectors of Real Symmetric Matrices , 1963, JACM.

[152]  Efstratios Gallopoulos,et al.  Computing smallest singular triplets with implicitly restarted Lanczos bidiagonalization , 2004, Applied Numerical Mathematics.

[153]  P. Lancaster,et al.  Factorization of selfadjoint matrix polynomials with constant signature , 1982 .

[154]  James Demmel,et al.  LAPACK Users' Guide, Third Edition , 1999, Software, Environments and Tools.

[155]  Axel Ruhe,et al.  Rational Krylov: A Practical Algorithm for Large Sparse Nonsymmetric Matrix Pencils , 1998, SIAM J. Sci. Comput..

[156]  H. P. Kempen On the quadratic convergence of the special cyclic Jacobi method , 1966 .

[157]  V. V. Voyevodin,et al.  A method for the solution of the complete eigenvalue problem , 1963 .

[158]  David S. Scott,et al.  Implementing Lanczos-like algorithms on hypercube architectures , 1989 .

[159]  I. Schur Über die charakteristischen Wurzeln einer linearen Substitution mit einer Anwendung auf die Theorie der Integralgleichungen , 1909 .

[160]  P. Hager,et al.  The Rational Krylov Algorithm for Nonlinear Eigenvalue Problems, in Computational Mechanics for the Twenty-First Century , 2000 .

[161]  M. C. Jordan Traite des substitutions et des equations algebriques , 1870 .

[162]  W. Givens Numerical Computation of the Characteristic Values of a Real Symmetric Matrix , 1954 .

[163]  Chao Yang,et al.  Solving Large-scale Eigenvalue Problems in SciDAC Applications , 2005 .

[164]  Karen S. Braman,et al.  The Multi-shift Qr-algorithm: Aggressive Deeation, Maintaining Well Focused Shifts, and Level 3 Performance , 1999 .

[165]  O. Taussky A Recurring Theorem on Determinants , 1949 .

[166]  R. T. Gregory Computing eigenvalues and eigenvectors of a symmetric matrix on the ILLIAC , 1953 .

[167]  Jean Descloux,et al.  Bounds for the spectral norm of functions of matrices , 1963 .

[168]  E. E. Osborne On Pre-Conditioning of Matrices , 1960, JACM.

[169]  David S. Watkins,et al.  Convergence of algorithms of decomposition type for the eigenvalue problem , 1991 .

[170]  J. Cullum,et al.  A generalized nonsymmetric Lanczos procedure , 1989 .

[171]  W. Arnoldi The principle of minimized iterations in the solution of the matrix eigenvalue problem , 1951 .

[172]  M. Lotkin Characteristic values of arbitrary matrices , 1956, ACM '56.

[173]  James Hardy Wilkinson,et al.  The Calculation of the Eigenvectors of Codiagonal Matrices , 1958, Comput. J..

[174]  T. D. Roopamala,et al.  Eigenvalues of Structural Matrices Via Gerschgorin Theorem , 2010 .

[175]  P. Hager Eigenfrequency Analysis FE-Adaptivity and a Nonlinear Eigenproblem Algorithm , 2001 .

[176]  C. Lanczos An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[177]  G. Stewart,et al.  An Algorithm for Computing Reducing Subspaces by Block Diagonalization. , 1979 .

[178]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[179]  Maurice Clint,et al.  The Evaluation of Eigenvalues and Eigenvectors of Real Symmetric Matrices by Simultaneous Iteration , 1970, Comput. J..

[180]  G. W. Stewart,et al.  Matrix algorithms , 1998 .

[181]  David S. Watkins,et al.  Forward Stability and Transmission of Shifts in the QR Algorithm , 1995, SIAM J. Matrix Anal. Appl..

[182]  Chandler Davis Explicit functional calculus , 1973 .