Randomized matrix-free trace and log-determinant estimators

We present randomized algorithms for estimating the trace and determinant of Hermitian positive semi-definite matrices. The algorithms are based on subspace iteration, and access the matrix only through matrix vector products. We analyse the error due to randomization, for starting guesses whose elements are Gaussian or Rademacher random variables. The analysis is cleanly separated into a structural (deterministic) part followed by a probabilistic part. Our absolute bounds for the expectation and concentration of the estimators are non-asymptotic and informative even for matrices of low dimension. For the trace estimators, we also present asymptotic bounds on the number of samples (columns of the starting guess) required to achieve a user-specified relative error. Numerical experiments illustrate the performance of the estimators and the tightness of the bounds on low-dimensional matrices, and on a challenging application in uncertainty quantification arising from Bayesian optimal experimental design.

[1]  G. Wahba.  Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy , 1977 .

[2]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[3]  Diane Valérie Ouellette.  Schur complements and statistics , 1981 .

[4]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[5]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[6]  G. Wahba.  Spline models for observational data , 1990 .

[7]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[8]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[9]  W. Näther.  Optimum experimental designs , 1994 .

[10]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[11]  G. Golub,et al.  Some large-scale matrix computation problems , 1996 .

[12]  G. Golub,et al.  Generalized cross-validation for large scale problems , 1997 .

[13]  M. Ledoux.  On Talagrand's deviation inequalities for product measures , 1997 .

[14]  Ronald P. Barry,et al.  Monte Carlo estimates of the log determinant of large sparse matrices , 1999 .

[15]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[16]  D. Ucinski.  Optimal measurement methods for distributed parameter system identification , 2004 .

[17]  James P. LeSage,et al.  Chebyshev approximation of log-determinants of spatial weight matrices , 2004, Comput. Stat. Data Anal..

[18]  Eli Upfal,et al.  Probability and Computing: Randomized Algorithms and Probabilistic Analysis , 2005 .

[19]  George Biros,et al.  DYNAMIC DATA-DRIVEN INVERSION FOR TERASCALE SIMULATIONS: REAL-TIME IDENTIFICATION OF AIRBORNE CONTAMINANTS , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[20]  Per-Gunnar Martinsson,et al.  Randomized algorithms for the low-rank approximation of matrices , 2007, Proceedings of the National Academy of Sciences.

[21]  M. Rudelson,et al.  The smallest singular value of a random rectangular matrix , 2008, 0802.3956.

[22]  Yunong Zhang,et al.  Log-det approximation based on uniformly distributed seeds and its application to Gaussian process regression , 2008 .

[23]  E. Haber,et al.  Numerical methods for experimental design of large-scale linear ill-posed inverse problems , 2008 .

[24]  D. Stirling.  Mathematical Analysis and Proof , 2009 .

[25]  C. Chui,et al.  Article in Press Applied and Computational Harmonic Analysis a Randomized Algorithm for the Decomposition of Matrices , 2022 .

[26]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[27]  Joel A. Tropp,et al.  Improved Analysis of the subsampled Randomized Hadamard Transform , 2010, Adv. Data Sci. Adapt. Anal..

[28]  N. Petra,et al.  Model Variational Inverse Problems Governed by Partial Differential Equations , 2011 .

[29]  Y. Saad.  Numerical Methods for Large Eigenvalue Problems , 2011 .

[30]  Y. Saad,et al.  Numerical Methods for Large Eigenvalue Problems , 2011 .

[31]  Mihai Anitescu,et al.  Computing f(A)b via Least Squares Polynomial Approximations , 2011, SIAM J. Sci. Comput..

[32]  Sivan Toledo,et al.  Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix , 2011, JACM.

[33]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[34]  Bart G. van Bloemen Waanders,et al.  Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations , 2011, SIAM J. Sci. Comput..

[35]  Luis Tenorio,et al.  Numerical methods for A-optimal designs with a sparsity constraint for ill-posed inverse problems , 2012, Comput. Optim. Appl..

[36]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[37]  MIHAI ANITESCU,et al.  A Matrix-free Approach for Solving the Parametric Gaussian Process Maximum Likelihood Problem , 2012, SIAM J. Sci. Comput..

[38]  Yousef Saad,et al.  A Probing Method for Computing the Diagonal of the Matrix Inverse ∗ , 2010 .

[39]  Georg Stadler,et al.  A-Optimal Design of Experiments for Infinite-Dimensional Bayesian Linear Inverse Problems with Regularized ℓ0-Sparsification , 2013, SIAM J. Sci. Comput..

[40]  O. Ghattas,et al.  On Bayesian A- and D-optimal experimental designs in infinite dimensions , 2014, 1408.6323.

[41]  A. Saibaba,et al.  Fast computation of uncertainty quantification measures in the geostatistical approach to solve inverse problems , 2014, 1404.1263.

[42]  Jinwoo Shin,et al.  Large-scale log-determinant computation through stochastic Chebyshev expansions , 2015, ICML.

[43]  Joel A. Tropp,et al.  An Introduction to Matrix Concentration Inequalities , 2015, Found. Trends Mach. Learn..

[44]  Uri M. Ascher,et al.  Improved Bounds on Sample Size for Implicit Matrix Trace Estimators , 2013, Found. Comput. Math..

[45]  Ming Gu,et al.  Subspace Iteration Randomization and Singular Value Problems , 2014, SIAM J. Sci. Comput..

[46]  Christos Boutsidis,et al.  A Randomized Algorithm for Approximating the Log Determinant of a Symmetric Positive Definite Matrix , 2015, ArXiv.

[47]  Danny C. Sorensen,et al.  A DEIM Induced CUR Factorization , 2014, SIAM J. Sci. Comput..

[48]  Michael W. Mahoney,et al.  Revisiting the Nystrom Method for Improved Large-scale Machine Learning , 2013, J. Mach. Learn. Res..

[49]  Lin Lin,et al.  Randomized estimation of spectral densities of large matrices made accurate , 2015, Numerische Mathematik.