Singular vector and singular subspace distribution for the matrix denoising model

In this paper, we study the matrix denosing model $Y=S+X$, where $S$ is a low-rank deterministic signal matrix and $X$ is a random noise matrix, and both are $M\times n$. In the scenario that $M$ and $n$ are comparably large and the signals are supercritical, we study the fluctuation of the outlier singular vectors of $Y$. More specifically, we derive the limiting distribution of angles between the principal singular vectors of $Y$ and their deterministic counterparts, the singular vectors of $S$. Further, we also derive the distribution of the distance between the subspace spanned by the principal singular vectors of $Y$ and that spanned by the singular vectors of $S$. It turns out that the limiting distributions depend on the structure of the singular vectors of $S$ and the distribution of $X$, and thus they are non-universal.

[1]  Jianhua Z. Huang,et al.  Biclustering via Sparse Singular Value Decomposition , 2010, Biometrics.

[2]  A. Singer.  Angular Synchronization by Eigenvectors and Semidefinite Programming. , 2009, Applied and computational harmonic analysis.

[3]  Alex Bloemendal,et al.  Limits of spiked random matrices I , 2010, Probability Theory and Related Fields.

[4]  Ruth M. Pfeiffer,et al.  On the distribution of the left singular vectors of a random matrix and its applications , 2008 .

[5]  N. Pillai,et al.  Universality of covariance matrices , 2011, 1110.2501.

[6]  Miguel Á. Carreira-Perpiñán,et al.  A Denoising View of Matrix Completion , 2011, NIPS.

[7]  Steven Kay,et al.  Fundamentals Of Statistical Signal Processing , 2001 .

[8]  Vladimir Koltchinskii,et al.  Normal approximation and concentration of spectral projectors of sample covariance , 2015, 1504.07333.

[9]  Vladimir Koltchinskii,et al.  Asymptotics and Concentration Bounds for Bilinear Forms of Spectral Projectors of Sample Covariance , 2014, 1408.4643.

[10]  C. Donati-Martin,et al.  Non universality of fluctuations of outlier eigenvectors for block diagonal deformations of Wigner matrices , 2018, 1807.07773.

[11]  C. Stein.  Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[12]  Alex Bloemendal,et al.  Limits of spiked random matrices II , 2011, 1109.3704.

[13]  S. Péché.  The largest eigenvalue of small rank perturbations of Hermitian random matrices , 2004, math/0411487.

[14]  J. Lee,et al.  Tracy-Widom Distribution for the Largest Eigenvalue of Real Sample Covariance Matrices with General Population , 2014, 1409.4979.

[15]  D. Paul.  ASYMPTOTICS OF SAMPLE EIGENSTRUCTURE FOR A LARGE DIMENSIONAL SPIKED COVARIANCE MODEL , 2007 .

[16]  David E. Tyler.  Asymptotic Inference for Eigenvectors , 1981 .

[17]  Yiqiao Zhong,et al.  Optimal Subspace Estimation Using Overidentifying Vectors via Generalized Method of Moments , 2018, 1805.02826.

[18]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[19]  Jun Yin,et al.  The Isotropic Semicircle Law and Deformation of Wigner Matrices , 2011, 1110.6449.

[20]  J. W. Silverstein,et al.  Eigenvalues of large sample covariance matrices of spiked population models , 2004, math/0408165.

[21]  Raj Rao Nadakuditi,et al.  The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices , 2009, 0910.2120.

[22]  Raj Rao Nadakuditi,et al.  The singular values and vectors of low rank perturbations of large rectangular random matrices , 2011, J. Multivar. Anal..

[23]  C. Donati-Martin,et al.  Spectrum of deformed random matrices and free probability , 2016, 1607.05560.

[24]  Nicolas Boumal,et al.  Near-Optimal Bounds for Phase Synchronization , 2017, SIAM J. Optim..

[25]  C. Donati-Martin,et al.  The largest eigenvalues of finite rank deformation of large Wigner matrices: Convergence and nonuniversality of the fluctuations. , 2007, 0706.0136.

[26]  Jiang Hu,et al.  Canonical correlation coefficients of high-dimensional Gaussian vectors: Finite rank case , 2014, The Annals of Statistics.

[27]  M. Gavish,et al.  Multidimensional scaling of noisy high dimensional data , 2018, Applied and Computational Harmonic Analysis.

[28]  Boris A. Khoruzhenko,et al.  Asymptotic properties of large random matrices with independent entries , 1996 .

[29]  V. Vu,et al.  Random perturbation of low rank matrices: Improving classical bounds , 2013, 1311.2657.

[30]  Helmut Bölcskei,et al.  Dimensionality-reduced subspace clustering , 2015, ArXiv.

[31]  Anat Levin,et al.  Natural image denoising: Optimality and inherent bounds , 2011, CVPR 2011.

[32]  Jun Yin,et al.  Anisotropic local laws for random matrices , 2014, 1410.3516.

[33]  Anru R. Zhang,et al.  Rate-Optimal Perturbation Bounds for Singular Subspaces with Applications to High-Dimensional Statistics , 2016, 1605.00353.

[34]  Dong Xia,et al.  Confidence interval of singular vectors for high-dimensional and low-rank matrix regression , 2018, ArXiv.

[35]  Xiucai Ding,et al.  Principal components of spiked covariance matrices in the supercritical regime , 2019, 1907.12251.

[36]  Dan Yang,et al.  Rate Optimal Denoising of Simultaneously Sparse and Low Rank Matrices , 2014, J. Mach. Learn. Res..

[37]  Qiang Sun,et al.  Modified Multidimensional Scaling and High Dimensional Clustering , 2018, ArXiv.

[38]  Fan Yang,et al.  Spiked separable covariance matrices and principal components , 2019, 1905.13060.

[39]  Philippe Loubaton,et al.  A subspace estimator for fixed rank perturbations of large random matrices , 2011, J. Multivar. Anal..

[40]  D. Féral,et al.  The Largest Eigenvalue of Rank One Deformation of Large Wigner Matrices , 2006, math/0605624.

[41]  A. Guionnet,et al.  Fluctuations of the Extreme Eigenvalues of Finite Rank Deformations of Random Matrices , 2010, 1009.0145.

[42]  A. Singer,et al.  Orientability and Diffusion Maps. , 2011, Applied and computational harmonic analysis.

[43]  Weichen Wang,et al.  An $\ell_{\infty}$ Eigenvector Perturbation Bound and Its Application , 2017, J. Mach. Learn. Res..

[44]  Jun Yin,et al.  The outliers of a deformed Wigner matrix , 2012, 1207.5619.

[45]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[46]  Iain M. Johnstone,et al.  Spiked covariances and principal components analysis in high-dimensional random effects models , 2018, 1806.09529.

[47]  Wang Zhou,et al.  Universality for the largest eigenvalue of sample covariance matrices with general population , 2013, 1304.5690.

[48]  C. Priebe,et al.  Signal‐plus‐noise matrix models: eigenvector deviations and fluctuations , 2018, Biometrika.

[49]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[50]  Xiucai Ding,et al.  High dimensional deformed rectangular matrices with applications in matrix denoising , 2017, Bernoulli.

[51]  H. Yau,et al.  Isotropic local laws for sample covariance and generalized Wigner matrices , 2013, 1308.5729.

[52]  M. Capitaine.  Limiting eigenvectors of outliers for Spiked Information-Plus-Noise type matrices , 2017, 1701.08069.

[53]  C. Donati-Martin,et al.  Central limit theorems for eigenvalues of deformations of Wigner matrices , 2009, 0903.4740.

[54]  Jonathan E. Taylor,et al.  Inference for eigenvalues and eigenvectors of Gaussian symmetric matrices , 2008, 0901.3290.

[55]  Garvesh Raskutti,et al.  Convex regularization for high-dimensional multiresponse tensor regression , 2015, The Annals of Statistics.

[56]  L. Pastur,et al.  CENTRAL LIMIT THEOREM FOR LINEAR EIGENVALUE STATISTICS OF RANDOM MATRICES WITH INDEPENDENT ENTRIES , 2008, 0809.4698.

[57]  Kevin Schnelli,et al.  Local law and Tracy–Widom limit for sparse random matrices , 2016, 1605.08767.

[58]  Jianfeng Yao,et al.  On sample eigenvalues in a generalized spiked population model , 2008, J. Multivar. Anal..

[59]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[60]  D. Donoho,et al.  Minimax risk of matrix denoising by singular value thresholding , 2013, 1304.2085.

[61]  J. R. Schott.  Some tests for common principal component subspaces in several groups , 1991 .

[62]  Vincent Q. Vu,et al.  MINIMAX SPARSE PRINCIPAL SUBSPACE ESTIMATION IN HIGH DIMENSIONS , 2012, 1211.0373.

[63]  Z. Bai,et al.  Central limit theorems for eigenvalues in a spiked population model , 2008, 0806.2503.

[64]  Reza Javidan,et al.  MUSIC algorithm for DOA estimation using MIMO arrays , 2011, 2011 6th International Conference on Telecommunication Systems, Services, and Applications (TSSA).

[65]  Boaz Nadler,et al.  ON THE DISTRIBUTION OF ROY'S LARGEST ROOT TEST IN MANOVA AND IN SIGNAL DETECTION IN NOISE , 2011 .

[66]  Sara van de Geer,et al.  De-biased sparse PCA: Inference and testing for eigenstructure of large covariance matrices , 2018, 1801.10567.

[67]  Antti Knowles,et al.  Averaging Fluctuations in Resolvents of Random Band Matrices , 2012, 1205.5664.

[68]  A. Soshnikov,et al.  On finite rank deformations of Wigner matrices , 2011, 1103.3731.

[69]  T. W. Anderson.  ASYMPTOTIC THEORY FOR PRINCIPAL COMPONENT ANALYSIS , 1963 .

[70]  S. Péché,et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices , 2004, math/0403022.

[71]  Noureddine El Karoui.  Tracy–Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices , 2005, math/0503109.

[72]  Qiang Sun,et al.  Principal Component Analysis for Big Data , 2018, Wiley StatsRef: Statistics Reference Online.

[73]  Jianqing Fan,et al.  An l∞ Eigenvector Perturbation Bound and Its Application to Robust Covariance Estimation , 2018, Journal of machine learning research : JMLR.

[74]  A. Robert Calderbank,et al.  The Role of Principal Angles in Subspace Classification , 2015, IEEE Transactions on Signal Processing.

[75]  H. Yau,et al.  On the principal components of sample covariance matrices , 2014, 1404.0788.

[76]  C. Priebe,et al.  The two-to-infinity norm and singular subspace geometry with applications to high-dimensional statistics , 2017, The Annals of Statistics.

[77]  Vladimir Koltchinskii,et al.  Efficient estimation of linear functionals of principal components , 2017, The Annals of Statistics.