Weak Detection of Signal in the Spiked Wigner Model

We consider the problem of detecting the presence of signal in a rank-one signal-plus-noise data matrix. In case the signal-to-noise ratio is under the threshold below which a reliable detection is impossible, we propose a hypothesis test based on the linear spectral statistics of the data matrix. The error of the proposed test is optimal as it matches the error of the likelihood ratio test that minimizes the sum of the Type-I and Type-II errors. The test is data-driven and does not depend on the distribution of the signal or the noise. If the density of the noise is known, it can be further improved by an entrywise transformation of the data matrix to lower the error of the test. As an intermediate step, we establish a central limit theorem for the linear spectral statistics of general rank-one spiked Wigner matrices.

[1]  B. Nadler.  Finite sample approximation results for principal component analysis: a matrix perturbation approach , 2009, 0901.3245.

[2]  J. Baik,et al.  Ferromagnetic to Paramagnetic Transition in Spherical Spin Glass , 2018, Journal of Statistical Physics.

[3]  Z. Bai,et al.  CLT for linear spectral statistics of large dimensional sample covariance matrices with dependent data , 2017, Statistical Papers.

[4]  Jun Yin,et al.  The Isotropic Semicircle Law and Deformation of Wigner Matrices , 2011, 1110.6449.

[5]  Michael I. Jordan,et al.  Fundamental limits of detection in the spiked Wigner model , 2018, 1806.09588.

[6]  Marc Lelarge,et al.  Fundamental limits of symmetric low-rank matrix estimation , 2016, Probability Theory and Related Fields.

[7]  T. Tao,et al.  Random Matrices: Universality of Local Eigenvalue Statistics up to the Edge , 2009, 0908.1982.

[8]  Yu. I. Ingster,et al.  Detection of a sparse submatrix of a high-dimensional noisy matrix , 2011, 1109.0898.

[9]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[10]  Kevin Schnelli,et al.  Local law and Tracy–Widom limit for sparse random matrices , 2016, 1605.08767.

[11]  Raj Rao Nadakuditi,et al.  The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices , 2009, 0910.2120.

[12]  Ronald F. Boisvert,et al.  NIST Handbook of Mathematical Functions , 2010 .

[13]  I. Johnstone,et al.  On Consistency and Sparsity for Principal Components Analysis in High Dimensions , 2009, Journal of the American Statistical Association.

[14]  C. Tracy,et al.  Mathematical Physics © Springer-Verlag 1996 On Orthogonal and Symplectic Matrix Ensembles , 1995 .

[15]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[16]  D. Paul.  ASYMPTOTICS OF SAMPLE EIGENSTRUCTURE FOR A LARGE DIMENSIONAL SPIKED COVARIANCE MODEL , 2007 .

[17]  Jianfeng Yao,et al.  On the convergence of the spectral empirical process of Wigner matrices , 2005 .

[18]  J. Baik,et al.  Free energy of bipartite spherical Sherrington–Kirkpatrick model , 2017, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques.

[19]  S. Péché,et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices , 2004, math/0403022.

[20]  J. Baik,et al.  Fluctuations of the Free Energy of the Spherical Sherrington–Kirkpatrick Model , 2015, Journal of Statistical Physics.

[21]  H. Yau,et al.  Rigidity of eigenvalues of generalized Wigner matrices , 2010, 1007.4652.

[22]  I. Johnstone,et al.  Testing in high-dimensional spiked models , 2015, The Annals of Statistics.

[23]  J. Baik,et al.  Fluctuations of the Free Energy of the Spherical Sherrington–Kirkpatrick Model with Ferromagnetic Interaction , 2016, Annales Henri Poincaré.

[24]  D. Féral,et al.  The Largest Eigenvalue of Rank One Deformation of Large Wigner Matrices , 2006, math/0605624.

[25]  S. Péché.  The largest eigenvalue of small rank perturbations of Hermitian random matrices , 2004, math/0411487.

[26]  Marcelo J. Moreira,et al.  Asymptotic power of sphericity tests for high-dimensional data , 2013, 1306.4867.

[27]  J. Lee,et al.  Tracy-Widom Distribution for the Largest Eigenvalue of Real Sample Covariance Matrices with General Population , 2014, 1409.4979.

[28]  Zongming Ma,et al.  Optimal hypothesis testing for stochastic block models with growing degrees , 2017, ArXiv.

[29]  Universality for general Wigner-type matrices , 2015, 1506.05098.

[30]  C. Donati-Martin,et al.  The largest eigenvalues of finite rank deformation of large Wigner matrices: Convergence and nonuniversality of the fluctuations. , 2007, 0706.0136.

[31]  Alexei Onatski,et al.  Signal detection in high dimension: The multispiked case , 2012, 1210.5663.

[32]  Ankur Moitra,et al.  Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models , 2018, The Annals of Statistics.

[33]  Andrea Montanari,et al.  On the Limitation of Spectral Methods: From the Gaussian Hidden Clique Problem to Rank One Perturbations of Gaussian Tensors , 2014, IEEE Transactions on Information Theory.

[34]  Nicolas Macris,et al.  Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula , 2016, NIPS.