Universality for Eigenvalue Algorithms on Sample Covariance Matrices

We prove a universal limit theorem for the halting time, or iteration count, of the power/inverse power methods and the QR eigenvalue algorithm. Specifically, we analyze the required number of iterations to compute extreme eigenvalues of random, positive definite sample covariance matrices to within a prescribed tolerance. The universality theorem provides a complexity estimate for the algorithms which, in this random setting, holds with high probability. The method of proof relies on recent results on the statistics of the eigenvalues and eigenvectors of random sample covariance matrices (i.e., delocalization, rigidity, and edge universality).

[1]  Y. Yin.  Limiting spectral distribution for a class of random matrices , 1986 .

[2]  P. Deift,et al.  On the condition number of the critically-scaled Laguerre Unitary Ensemble , 2015, 1507.00750.

[3]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[4]  G. Pan,et al.  On asymptotics of eigenvectors of large sample covariance matrix , 2007, 0708.1720.

[5]  Rüdiger Reischuk,et al.  Smoothed analysis of binary search trees , 2005, Theor. Comput. Sci..

[6]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[7]  D Teng.  Smoothed Analysis of Algorithms , 2002 .

[8]  J. W. Silverstein.  The Smallest Eigenvalue of a Large Dimensional Wishart Matrix , 1985 .

[9]  Thomas Trogdon,et al.  Smoothed Analysis for the Conjugate Gradient Algorithm , 2016, 1605.06438.

[10]  P. J. Forrester,et al.  Joint distribution of the first and second eigenvalues at the soft edge of unitary ensembles , 2012, 1209.2190.

[11]  E. Kostlan.  Complexity theory of numerical linear algebra , 1988 .

[12]  J. Ramírez,et al.  Beta ensembles, stochastic Airy spectrum, and a diffusion , 2006, math/0607331.

[13]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[14]  K. Wachter.  The Strong Limits of Random Matrix Spectra for Sample Matrices of Independent Elements , 1978 .

[15]  K. Johansson.  Shape Fluctuations and Random Matrices , 1999, math/9903134.

[16]  P. Deift,et al.  How long does it take to compute the eigenvalues of a random, symmetric matrix? , 2012, 1203.4635.

[17]  J. Neumann,et al.  Numerical inverting of matrices of high order. II , 1951 .

[18]  P. Deift,et al.  Universality for the Toda Algorithm to Compute the Largest Eigenvalue of a Random Matrix , 2016, 1604.07384.

[19]  P. Forrester.  The spectrum edge of random matrix ensembles , 1993 .

[20]  N. Pillai,et al.  Universality of covariance matrices , 2011, 1110.2501.

[21]  Shang-Hua Teng,et al.  Smoothed Analysis of the Condition Numbers and Growth Factors of Matrices , 2003, SIAM J. Matrix Anal. Appl..

[22]  David S. Watkins.  Isospectral Flows , 1996 .

[23]  Yann LeCun,et al.  Universality in halting time and its applications in optimization , 2015, ArXiv.

[24]  S. Smale.  On the efficiency of algorithms of analysis , 1985 .

[25]  H. Yau,et al.  On the principal components of sample covariance matrices , 2014, 1404.0788.

[26]  P. Deift,et al.  Universality in numerical computations with random data , 2014, Proceedings of the National Academy of Sciences.

[27]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[28]  J. W. Silverstein.  Eigenvalues and eigenvectors of large di-mensional sample covariance matrices , 1986 .

[29]  P. Deift,et al.  Universality for the Toda algorithm to compute the eigenvalues of a random matrix , 2016 .

[30]  S. Péché,et al.  Universality of local eigenvalue statistics for some sample covariance matrices , 2005 .

[31]  J. G. F. Francis,et al.  The QR Transformation A Unitary Analogue to the LR Transformation - Part 1 , 1961, Comput. J..

[32]  P. Deift,et al.  Ordinary differential equations and the symmetric eigenvalue problem , 1983 .

[33]  K. Borgwardt.  The Simplex Method: A Probabilistic Analysis , 1986 .