Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence

We consider two alternative tests to the Higher Criticism test of Donoho and Jin [Ann. Statist. 32 (2004) 962-994] for high-dimensional means under the sparsity of the nonzero means for sub-Gaussian distributed data with unknown column-wise dependence. The two alternative test statistics are constructed by first thresholding $L_1$ and $L_2$ statistics based on the sample means, respectively, followed by maximizing over a range of thresholding levels to make the tests adaptive to the unknown signal strength and sparsity. The two alternative tests can attain the same detection boundary of the Higher Criticism test in [Ann. Statist. 32 (2004) 962-994] which was established for uncorrelated Gaussian data. It is demonstrated that the maximal $L_2$-thresholding test is at least as powerful as the maximal $L_1$-thresholding test, and both the maximal $L_2$ and $L_1$-thresholding tests are at least as powerful as the Higher Criticism test.

[1]  H. Joe.  Multivariate models and dependence concepts , 1998 .

[2]  D. Pollard,et al.  An introduction to functional central limit theorems for dependent stochastic processes , 1994 .

[3]  Yihong Wu,et al.  Optimal Detection For Sparse Mixtures , 2012, ArXiv.

[4]  Q. Shao,et al.  TOWARDS A UNIVERSAL SELF-NORMALIZED MODERATE DEVIATION , 2008 .

[5]  P. Hall,et al.  PROPERTIES OF HIGHER CRITICISM UNDER STRONG DEPENDENCE , 2008, 0803.2095.

[6]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[7]  V. V. Petrov.  Limit Theorems of Probability Theory: Sequences of Independent Random Variables , 1995 .

[8]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[9]  G. Pisier.  Some applications of the metric entropy condition to harmonic analysis , 1983 .

[10]  P. Doukhan.  Mixing: Properties and Examples , 1994 .

[11]  Song-xi Chen,et al.  A two-sample test for high-dimensional data with applications to gene-set testing , 2010, 1002.4547.

[12]  Jiashun Jin,et al.  Optimal detection of heterogeneous and heteroscedastic mixtures , 2011 .

[13]  Masaaki Sibuya,et al.  Bivariate extreme statistics, I , 1960 .

[14]  Jiashun Jin,et al.  Robustness and accuracy of methods for high dimensional data analysis based on Student's t‐statistic , 2010, 1001.3886.

[15]  D. Donoho,et al.  Higher criticism thresholding: Optimal feature selection when useful features are rare and weak , 2008, Proceedings of the National Academy of Sciences.

[16]  R. C. Bradley.  Basic properties of strong mixing conditions. A survey and some open questions , 2005, math/0511078.

[17]  G. Lugosi,et al.  Detecting Positive Correlations in a Multivariate Sample , 2012, 1202.5536.

[18]  Aurore Delaigle,et al.  Higher Criticism in the Context of Unknown Distribution, Non-independence and Classification , 2009 .

[19]  J. Hüsler.  Extremes and related properties of random sequences and processes , 1984 .

[20]  Q. Shao.  Self-normalized large deviations , 1997 .

[21]  G. Lugosi,et al.  Detection of correlations , 2011, 1106.1193.

[22]  P. Hall,et al.  RELATIVE ERRORS IN CENTRAL LIMIT THEOREMS FOR STUDENT'S t STATISTIC, WITH APPLICATIONS , 2009 .

[23]  D. Donoho,et al.  Higher criticism for detecting sparse heterogeneous mixtures , 2004, math/0410072.

[24]  Z. Bai,et al.  EFFECT OF HIGH DIMENSION: BY AN EXAMPLE OF A TWO SAMPLE PROBLEM , 1999 .

[25]  Jianqing Fan.  Test of Significance Based on Wavelet Thresholding and Neyman's Truncation , 1996 .

[26]  P. Hall,et al.  Innovated Higher Criticism for Detecting Sparse Signals in Correlated Noise , 2009, 0902.3837.