Inference for linear forms of eigenvectors under minimal eigenvalue separation: Asymmetry and heteroscedasticity

A fundamental task that spans numerous applications is inference and uncertainty quantification for linear functionals of the eigenvectors of an unknown low-rank matrix. We prove that this task can be accomplished in a setting where the true matrix is symmetric and the additive noise matrix contains independent (and non-symmetric) entries. Specifically, we develop algorithms that produce confidence intervals for linear forms of individual eigenvectors, based on eigen-decomposition of the asymmetric data matrix followed by a careful de-biasing scheme. The proposed procedures and the accompanying theory enjoy several important features: (1) distribution-free (i.e. prior knowledge about the noise distributions is not needed); (2) adaptive to heteroscedastic noise; (3) statistically optimal under Gaussian noise. Along the way, we establish procedures to construct optimal confidence intervals for the eigenvalues of interest. All this happens under minimal eigenvalue separation, a condition that goes far beyond what generic matrix perturbation theory has to offer. Our studies fall under the category of “fine-grained” functional inference in low-complexity models.

[1]  Yuxin Chen,et al.  Compressive Two-Dimensional Harmonic Retrieval via Atomic Norm Minimization , 2015, IEEE Transactions on Signal Processing.

[2]  Jianqing Fan,et al.  Asymptotic Theory of Eigenvectors for Large Random Matrices , 2019, 1902.06846.

[3]  Anru R. Zhang,et al.  Rate-Optimal Perturbation Bounds for Singular Subspaces with Applications to High-Dimensional Statistics , 2016, 1605.00353.

[4]  Adel Javanmard,et al.  Confidence intervals and hypothesis testing for high-dimensional regression , 2013, J. Mach. Learn. Res..

[5]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[6]  Adel Javanmard,et al.  Localization from Incomplete Noisy Distance Measurements , 2011, Foundations of Computational Mathematics.

[7]  Ming Yuan,et al.  Statistical inferences of linear forms for noisy matrix completion , 2019, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[8]  Van H. Vu Singular vectors under random perturbation , 2011, Random Struct. Algorithms.

[9]  Florent Benaych-Georges,et al.  Outliers in the Single Ring Theorem , 2013, 1308.3064.

[10]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[11]  Rongrong Wang,et al.  Singular Vector Perturbation Under Gaussian Noise , 2012, SIAM J. Matrix Anal. Appl..

[12]  Chen Cheng,et al.  Asymmetry Helps: Eigenvalue and Eigenvector Analyses of Asymmetrically Perturbed Low-Rank Matrices , 2018, ArXiv.

[13]  Konstantin Tikhomirov,et al.  On delocalization of eigenvectors of random non-Hermitian matrices , 2018, Probability Theory and Related Fields.

[14]  Dong Xia,et al.  Perturbation of linear forms of singular vectors under Gaussian noise , 2015 .

[15]  Jun Yan,et al.  Adapting to Unknown Noise Distribution in Matrix Denoising , 2018, ArXiv.

[16]  I. Johnstone,et al.  Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model. , 2013, Annals of statistics.

[17]  V. Koltchinskii,et al.  Nuclear norm penalization and optimal rates for noisy low rank matrix completion , 2010, 1011.6256.

[18]  T. Tony Cai,et al.  Confidence intervals for high-dimensional linear regression: Minimax rates and adaptivity , 2015, 1506.05539.

[19]  Tapan K. Sarkar,et al.  Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise , 1990, IEEE Trans. Acoust. Speech Signal Process..

[20]  D. Donoho,et al.  Minimax risk of matrix denoising by singular value thresholding , 2013, 1304.2085.

[21]  Lloyd N. Trefethen,et al.  Generalizing Eigenvalue Theorems to Pseudospectra Theorems , 2001, SIAM J. Sci. Comput..

[22]  J. H. van Lint,et al.  Functions of one complex variable II , 1997 .

[23]  E. Brezin,et al.  NON-HERMITEAN DELOCALIZATION : MULTIPLE SCATTERING AND BOUNDS , 1998 .

[24]  Jianqing Fan,et al.  An l∞ Eigenvector Perturbation Bound and Its Application to Robust Covariance Estimation , 2018, Journal of machine learning research : JMLR.

[25]  Yuxin Chen,et al.  Robust Spectral Compressed Sensing via Structured Matrix Completion , 2013, IEEE Transactions on Information Theory.

[26]  Dong Xia Data-dependent Confidence Regions of Singular Subspaces , 2019, ArXiv.

[27]  Andrea Montanari,et al.  Matrix Completion from Noisy Entries , 2009, J. Mach. Learn. Res..

[28]  Chandler Davis The rotation of eigenvectors by a perturbation , 1963 .

[29]  R. Nickl,et al.  Uncertainty Quantification for Matrix Compressed Sensing and Quantum Tomography Problems , 2015, Progress in Probability.

[30]  Lydia T. Liu,et al.  $e$PCA: High dimensional exponential family PCA , 2016, The Annals of Applied Statistics.

[31]  S. Geer,et al.  On asymptotically optimal confidence regions and tests for high-dimensional models , 2013, 1303.0518.

[32]  Anru R. Zhang,et al.  Heteroskedastic PCA: Algorithm, optimality, and applications , 2018, The Annals of Statistics.

[33]  Mikhail Belkin,et al.  Unperturbed: spectral analysis beyond Davis-Kahan , 2017, ALT.

[34]  Jianqing Fan,et al.  ENTRYWISE EIGENVECTOR ANALYSIS OF RANDOM MATRICES WITH LOW EXPECTED RANK. , 2017, Annals of statistics.

[35]  A. Carpentier,et al.  Constructing confidence sets for the matrix completion problem , 2017, 1704.02760.

[36]  Junwei Lu,et al.  Inter-Subject Analysis: Inferring Sparse Interactions with Dense Intra-Graphs , 2017, 1709.07036.

[37]  V. Vu,et al.  Random perturbation of low rank matrices: Improving classical bounds , 2013, 1311.2657.

[38]  Leonidas J. Guibas,et al.  Near-Optimal Joint Object Matching via Convex Relaxation , 2014, ICML.

[39]  Yiqiao Zhong Eigenvector Under Random Perturbation: A Nonasymptotic Rayleigh-Schr\"{o}dinger Theory , 2017, 1702.00139.

[40]  A. Zee,et al.  Non-hermitian random matrix theory: Method of hermitian reduction , 1997 .

[41]  Yuling Yan,et al.  Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization , 2019, SIAM J. Optim..

[42]  Dong Xia,et al.  Confidence interval of singular vectors for high-dimensional and low-rank matrix regression , 2018, ArXiv.

[43]  Han Liu,et al.  A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models , 2014, 1412.8765.

[44]  Cun-Hui Zhang,et al.  Confidence intervals for low dimensional parameters in high dimensional linear models , 2011, 1110.2563.

[45]  Tengyao Wang,et al.  A useful variant of the Davis--Kahan theorem for statisticians , 2014, 1405.0680.

[46]  Daniel K Sodickson,et al.  Low‐rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components , 2015, Magnetic resonance in medicine.

[47]  A. Belloni,et al.  Inference for High-Dimensional Sparse Econometric Models , 2011, 1201.0220.

[48]  Yuxin Chen,et al.  Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview , 2018, IEEE Transactions on Signal Processing.

[49]  Alexandra Carpentier,et al.  On signal detection and confidence sets for low rank inference problems , 2015, 1507.03829.

[50]  Joel A. Tropp,et al.  An Introduction to Matrix Concentration Inequalities , 2015, Found. Trends Mach. Learn..

[51]  H. Vincent Poor,et al.  Subspace Estimation from Unbalanced and Incomplete Data Matrices: 𝓁2, ∞ Statistical Guarantees , 2021, ArXiv.

[52]  Martin Raič,et al.  Normal Approximation by Stein ’ s Method , 2003 .

[53]  Jianqing Fan,et al.  SIMPLE: Statistical inference on membership profiles in large networks , 2019, 1910.01734.

[54]  Yuxin Chen,et al.  Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution , 2017, Found. Comput. Math..

[55]  Martin J. Wainwright,et al.  Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees , 2015, ArXiv.

[56]  Sara van de Geer,et al.  De-biased sparse PCA: Inference and testing for eigenstructure of large covariance matrices , 2018, 1801.10567.

[57]  Yuxin Chen,et al.  Spectral Method and Regularized MLE Are Both Optimal for Top-$K$ Ranking , 2017, Annals of statistics.

[58]  J. Pauly,et al.  Accelerating parameter mapping with a locally low rank constraint , 2015, Magnetic resonance in medicine.

[59]  Yuling Yan,et al.  Inference and uncertainty quantification for noisy matrix completion , 2019, Proceedings of the National Academy of Sciences.

[60]  Ji Chen,et al.  Nonconvex Rectangular Matrix Completion via Gradient Descent Without ℓ₂,∞ Regularization , 2020, IEEE Transactions on Information Theory.

[61]  Yuxin Chen,et al.  The Projected Power Method: An Efficient Algorithm for Joint Alignment from Pairwise Differences , 2016, Communications on Pure and Applied Mathematics.

[62]  Andrea Montanari,et al.  The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning , 2018, The Annals of Statistics.

[63]  Anand Rajagopalan,et al.  Outlier Eigenvalue Fluctuations of Perturbed IID Matrices , 2015, 1507.01441.

[64]  Boris A Khoruzhenko LETTER TO THE EDITOR: Large- N eigenvalue distribution of randomly perturbed asymmetric matrices , 1996 .

[65]  Sommers,et al.  Spectrum of large random asymmetric matrices. , 1988, Physical review letters.

[66]  Martin J. Wainwright,et al.  High-Dimensional Statistics , 2019 .

[67]  Harrison H. Zhou,et al.  Asymptotic normality and optimalities in estimation of large Gaussian graphical models , 2013, 1309.6024.

[68]  Martin J. Wainwright,et al.  Value function estimation in Markov reward processes: Instance-dependent 𝓁∞-bounds for policy evaluation , 2019, ArXiv.

[69]  T. Tao Outliers in the spectrum of iid matrices with bounded rank perturbations , 2010 .

[70]  Martin J. Wainwright,et al.  Restricted strong convexity and weighted matrix completion: Optimal bounds with noise , 2010, J. Mach. Learn. Res..

[71]  George A. F. Seber,et al.  Linear regression analysis , 1977 .

[72]  B. Mehlig,et al.  EIGENVECTOR STATISTICS IN NON-HERMITIAN RANDOM MATRIX ENSEMBLES , 1998 .

[73]  P. Wedin Perturbation bounds in connection with singular value decomposition , 1972 .

[74]  Debashis Paul,et al.  PCA in High Dimensions: An Orientation , 2018, Proceedings of the IEEE.

[75]  C. Priebe,et al.  The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics , 2017, The Annals of Statistics.

[76]  S. Péché The largest eigenvalue of small rank perturbations of Hermitian random matrices , 2006 .