Universality for the Conjugate Gradient and MINRES Algorithms on Sample Covariance Matrices

We present a probabilistic analysis of two Krylov subspace methods for solving linear systems. We prove a central limit theorem for norms of the residual vectors that are produced by the conjugate gradient and MINRES algorithms when applied to a wide class of sample covariance matrices satisfying some standard moment conditions. The proof involves establishing a four moment theorem for the so-called spectral measure, implying, in particular, universality for the matrix produced by the Lanczos iteration. The central limit theorem then implies an almost-deterministic iteration count for the iterative methods in question.

[1]  A. Edelman,et al.  Matrix models for beta ensembles , 2002, math-ph/0206043.

[2]  Sean O'Rourke,et al.  Fluctuations of Matrix Entries of Regular Functions of Sample Covariance Random Matrices , 2011 .

[3]  Teodoro Collin.  RANDOM MATRIX THEORY , 2016 .

[4]  J. Wishart.  THE GENERALISED PRODUCT MOMENT DISTRIBUTION IN SAMPLES FROM A NORMAL MULTIVARIATE POPULATION , 1928 .

[5]  P. Deift,et al.  On the condition number of the critically-scaled Laguerre Unitary Ensemble , 2015, 1507.00750.

[6]  Thomas Trogdon,et al.  The conjugate gradient algorithm on well-conditioned Wishart matrices is almost deteriministic , 2019, Quarterly of Applied Mathematics.

[7]  Arno B. J. Kuijlaars,et al.  Superlinear Convergence of Conjugate Gradients , 2001, SIAM J. Numer. Anal..

[8]  K. Johansson.  On fluctuations of eigenvalues of random Hermitian matrices , 1998 .

[9]  Gérard Meurant.  On prescribing the convergence behavior of the conjugate gradient algorithm , 2019, Numerical Algorithms.

[10]  S. Szarek,et al.  Chapter 8 - Local Operator Theory, Random Matrices and Banach Spaces , 2001 .

[11]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[12]  P. Deift,et al.  Universality for the Toda Algorithm to Compute the Largest Eigenvalue of a Random Matrix , 2016, 1604.07384.

[13]  On spectral measures of random Jacobi matrices , 2016, 1601.01146.

[14]  Jun Yin,et al.  Anisotropic local laws for random matrices , 2014, 1410.3516.

[15]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[16]  P. Deift,et al.  Universality in numerical computations with random data , 2014, Proceedings of the National Academy of Sciences.

[17]  H. Yau,et al.  A Dynamical Approach to Random Matrix Theory , 2017 .

[18]  Fabian Pedregosa,et al.  Halting Time is Predictable for Large Models: A Universality Property and Average-Case Analysis , 2020, Foundations of Computational Mathematics.

[19]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[20]  Daniel A. Spielman.  The Smoothed Analysis of Algorithms , 2005, FCT.

[21]  J. W. Silverstein.  The Smallest Eigenvalue of a Large Dimensional Wishart Matrix , 1985 .

[22]  Arno B. J. Kuijlaars,et al.  Convergence Analysis of Krylov Subspace Iterations with Methods from Potential Theory , 2006, SIAM Rev..

[23]  Stephen Smale,et al.  On the average number of steps of the simplex method of linear programming , 1983, Math. Program..

[24]  Philipp Birken,et al.  Numerical Linear Algebra , 2011, Encyclopedia of Parallel Computing.

[25]  A. Edelman,et al.  Global spectrum fluctuations for the β-Hermite and β-Laguerre ensembles via matrix models , 2005, math-ph/0510043.

[26]  Shang-Hua Teng,et al.  Smoothed Analysis of the Condition Numbers and Growth Factors of Matrices , 2003, SIAM J. Matrix Anal. Appl..

[27]  Mariya Shcherbina,et al.  Central Limit Theorem for linear eigenvalue statistics of the Wigner and sample covariance random matrices , 2011, 1101.3249.

[28]  P. Deift.  Orthogonal Polynomials and Random Matrices: A Riemann-Hilbert Approach , 2000 .

[29]  Z. Bai,et al.  CLT for linear spectral statistics of large dimensional sample covariance matrices with dependent data , 2017, Statistical Papers.

[30]  T. Shirai,et al.  The mean spectral measures of random Jacobi matrices related to Gaussian beta ensembles , 2015, 1504.06904.

[31]  Thomas Trogdon,et al.  Universality for Eigenvalue Algorithms on Sample Covariance Matrices , 2017, SIAM J. Numer. Anal..

[32]  Archit Kulkarni,et al.  The Lanczos Algorithm Under Few Iterations: Concentration and Location of the Output , 2019, SIAM J. Matrix Anal. Appl..

[33]  K. Borgwardt.  The Simplex Method: A Probabilistic Analysis , 1986 .

[34]  On Fluctuations of Matrix Entries of Regular Functions of Wigner Matrices with Non-identically Distributed Entries , 2011, 1104.1663.

[35]  Thomas Trogdon,et al.  Smoothed Analysis for the Conjugate Gradient Algorithm , 2016, 1605.06438.

[36]  Ronald F. Boisvert,et al.  NIST Handbook of Mathematical Functions , 2010 .

[37]  J. Neumann,et al.  Numerical inverting of matrices of high order , 1947 .

[38]  Archit Kulkarni,et al.  The Lanczos Algorithm Under Few Iterations: Concentration and Location of the Ritz Values , 2019, ArXiv.

[39]  Thomas Trogdon,et al.  Universality in numerical computation with random data. Case studies, analytic results and some speculations , 2016 .

[40]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.