Inverse, Shifted Inverse, and Rayleigh Quotient Iteration as Newton's Method

The $l_2$ normalized inverse, shifted inverse, and Rayleigh quotient iterations are classic algorithms for approximating an eigenvector of a symmetric matrix. This work establishes rigorously that each iterate produced by one of these three algorithms can be viewed as a Newton's method iterate followed by a normalization. The equivalences given here are not meant to suggest changes to the implementations of the classic eigenvalue algorithms. However, they add further understanding to the formal structure of these iterations, and they provide an explanation for their good behavior despite the possible need to solve systems with nearly singular coefficient matrices. A historical development of these eigenvalue algorithms is presented. Using our equivalences and traditional Newton's method theory helps to gain understanding as to why normalized Newton's method, inverse iteration, and shifted inverse iteration are only linearly convergent and not quadratically convergent, as would be expected, and why a new l...

[1]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[2]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[3]  A. Ostrowski.  On the convergence of the Rayleigh quotient iteration for the computation of the characteristic roots and vectors. I , 1957 .

[4]  M. J. D. Powell,et al.  A fast algorithm for nonlinearly constrained optimization calculations , 1978 .

[5]  S. H. Crandall.  Iterative procedures related to relaxation methods for eigenvalue problems , 1951 .

[6]  F. Chatelin.  Simultaneous Newton’s Iteration for the Eigenproblem , 1984 .

[7]  R. Courant.  Variational methods for the solution of problems of equilibrium and vibrations , 1943 .

[8]  Ilse C. F. Ipsen.  Computing an Eigenvector with Inverse Iteration , 1997, SIAM Rev..

[9]  Anthony V. Fiacco,et al.  Nonlinear programming;: Sequential unconstrained minimization techniques , 1968 .

[10]  E. Pohlhausen,et al.  Berechnung der Eigenschwingungen statisch‐bestimmter Fachwerke , 1921 .

[11]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[12]  Ilse C. F. Ipsen.  A history of inverse iteration , 1994 .

[13]  Shih-Ping Han.  A globally convergent method for nonlinear programming , 1975 .

[14]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[15]  M. Gutzwiller,et al.  Moon-Earth-Sun: The oldest three-body problem , 1998 .

[16]  J. H. Wilkinson,et al.  Inverse Iteration, Ill-Conditioned Equations and Newton’s Method , 1979 .

[17]  R. Morgan,et al.  Generalizations of Davidson's method for computing eigenvalues of sparse symmetric matrices , 1986 .

[18]  R. Tapia.  On secant updates for use in general constrained optimization , 1988 .

[19]  H. Wielandt.  Das Iterationsverfahren bei nicht selbstadjungierten linearen Eigenwertaufgaben , 1944 .

[20]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[21]  Olvi L. Mangasarian,et al.  Superlinearly convergent quasi-newton algorithms for nonlinearly constrained optimization problems , 1976, Math. Program..

[22]  A. M. Ostrowski,et al.  On the convergence of the rayleigh quotient iteration for the computation of the characteristic roots and vectors. II , 1958 .

[23]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[24]  L. B. Rall,et al.  The solution of characteristic value-vector problems by Newton's method , 1968 .

[25]  Shih-Ping Han,et al.  Superlinearly convergent variable metric algorithms for general nonlinear programming problems , 1976, Math. Program..

[26]  Y. Saad,et al.  Numerical Methods for Large Eigenvalue Problems , 2011 .

[27]  A. C. Aitken.  XXV.—On Bernoulli's Numerical Solution of Algebraic Equations , 1927 .

[28]  Ilse C. F. Ipsen.  Helmut Wielandt's contributions to the numerical solution of complex eigenvalue problems , 1994 .

[29]  L. W.,et al.  The Theory of Sound , 1898, Nature.

[30]  L. Collatz.  Functional analysis and numerical mathematics , 1968 .

[31]  J. E. Dennis,et al.  On Newton-like methods , 1968 .

[32]  J. H. Wilkinson,et al.  IMPROVING THE ACCURACY OF COMPUTED EIGENVALUES AND EIGENVECTORS , 1983 .

[33]  R. A. Tapia,et al.  QUASI-NEWTON METHODS FOR EQUALITY CONSTRAINED OPTIMIZATION: EQUIVALENCE OF EXISTING METHODS AND A NEW IMPLEMENTATION , 1978 .

[34]  J. H. Wilkinson.  The algebraic eigenvalue problem , 1966 .

[35]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[36]  F. Chatelin.  ILL Conditioned Eigenproblems , 1986 .

[37]  James Hardy Wilkinson,et al.  The Calculation of the Eigenvectors of Codiagonal Matrices , 1958, Comput. J..

[38]  W. Kohn.  A Variational Iteration Method for Solving Secular Equations , 1949 .

[39]  Richard A. Tapia,et al.  The Projected Newton Method Has Order $1 + \sqrt 2 $ for the Symmetric Eigenvalue Problem , 1988 .

[40]  Gerard L. G. Sleijpen,et al.  A Jacobi-Davidson Iteration Method for Linear Eigenvalue Problems , 1996, SIAM J. Matrix Anal. Appl..

[41]  Richard A. Tapia,et al.  Origin and Evolution of the Secant Method in One Dimension , 2013, Am. Math. Mon..

[42]  M. J. D. Powell,et al.  THE CONVERGENCE OF VARIABLE METRIC METHODS FOR NONLINEARLY CONSTRAINED OPTIMIZATION CALCULATIONS , 1978 .