Detecting Positive Correlations in a Multivariate Sample

We consider the problem of testing whether a correlation matrix of a multivariate normal population is the identity matrix. We focus on sparse classes of alternatives where only a few entries are nonzero and, in fact, positive. We derive a general lower bound applicable to various classes and study the performance of some near-optimal tests. We pay special attention to computational feasibility and construct near-optimal tests that can be computed efficiently. Finally, we apply our results to prove new lower bounds for the clique number of high-dimensional random geometric graphs.

[1]  H. Nagao,et al.  On Some Test Criteria for Covariance Matrix , 1973 .

[2]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[3]  李幼升,et al.  Ph , 1989 .

[4]  M. Talagrand.  Concentration of measure and isoperimetric inequalities in product spaces , 1994, math/9406212.

[5]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[6]  Noga Alon,et al.  Finding a large hidden clique in a random graph , 1998, SODA '98.

[7]  Minimax detection of a signal for nondegenerate loss functions, and extremal convex problems , 1999 .

[8]  Olivier Ledoit,et al.  Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size , 2002 .

[9]  Y. Baraud.  Non-asymptotic minimax rates of testing in signal detection , 2002 .

[10]  Gábor Lugosi,et al.  Concentration Inequalities , 2008, COLT.

[11]  Jiashun Jin.  Detecting and estimating sparse mixtures , 2003 .

[12]  D. Donoho,et al.  Higher criticism for detecting sparse heterogeneous mixtures , 2004, math/0410072.

[13]  P. Rousseeuw,et al.  Wiley Series in Probability and Mathematical Statistics , 2005 .

[14]  Bart Funnekotter.  Oxford Oxford University , 2005 .

[15]  M. Srivastava.  Some Tests Concerning the Covariance Matrix in High Dimensional Data , 2005 .

[16]  James R. Schott,et al.  Testing for complete independence in high dimensions , 2005 .

[17]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[18]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[19]  Fanny Villers,et al.  Goodness-of-fit tests for high-dimensional Gaussian linear models , 2007, 0711.2119.

[20]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[21]  Noureddine El Karoui,et al.  Operator norm consistent estimation of large-dimensional sparse covariance matrices , 2008, 0901.3220.

[22]  P. Bickel,et al.  Regularized estimation of large covariance matrices , 2008, 0803.1909.

[23]  Carlos M. Carvalho,et al.  FLEXIBLE COVARIANCE ESTIMATION IN GRAPHICAL GAUSSIAN MODELS , 2008, 0901.3267.

[24]  E. Candès,et al.  Searching for a trail of evidence in a maze , 2007, math/0701668.

[25]  Jianqing Fan,et al.  Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation. , 2007, Annals of statistics.

[26]  P. Bickel,et al.  Covariance regularization by thresholding , 2009, 0901.3079.

[27]  Harrison H. Zhou,et al.  Optimal rates of convergence for covariance matrix estimation , 2010, 1010.3866.

[28]  Luc Devroye,et al.  Combinatorial Testing Problems , 2009, 0908.3437.

[29]  Song-xi Chen,et al.  Tests for High-Dimensional Covariance Matrices , 2010, Random Matrices: Theory and Applications.

[30]  P. Hall,et al.  Innovated Higher Criticism for Detecting Sparse Signals in Correlated Noise , 2009, 0902.3837.

[31]  N. Verzelen.  Minimax risks for sparse regressions: Ultra-high-dimensional phenomenons , 2010, 1008.0526.

[32]  G. Lugosi,et al.  High-dimensional random geometric graphs and their clique number , 2011 .

[33]  D. Richard Kuhn,et al.  COMBINATORIAL TESTING , 2011 .

[34]  G. Lugosi,et al.  Detection of correlations , 2011, 1106.1193.

[35]  Thomas J. Fisher.  On testing for an identity covariance matrix when the dimensionality equals or exceeds the sample size , 2012 .

[36]  P. Rigollet,et al.  Optimal detection of sparse principal components in high dimension , 2012, 1202.5070.

[37]  Philippe Rigollet,et al.  Computational Lower Bounds for Sparse PCA , 2013, ArXiv.

[38]  Yu. I. Ingster,et al.  Detection of a sparse submatrix of a high-dimensional noisy matrix , 2011, 1109.0898.

[39]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.