Testing mutual independence in high dimensions with sums of squares of rank correlations

We treat the problem of testing independence between m continuous variables when m can be larger than the available sample size n. We consider three types of test statistics that are constructed as sums or sums of squares of pairwise rank correlations. In the asymptotic regime where both m and n tend to infinity, a martingale central limit theorem is applied to show that the null distributions of these statistics converge to Gaussian limits, which are valid with no specific distributional or moment assumptions on the data. Using the framework of U-statistics, our result covers a variety of rank correlations including Kendall's tau and a dominating term of Spearman's rank correlation coefficient (rho), but also degenerate U-statistics such as Hoeffding's $D$, or the $\tau^*$ of Bergsma and Dassios (2014). As in the classical theory for U-statistics, the test statistics need to be scaled differently when the rank correlations used to construct them are degenerate U-statistics. The power of the considered tests is explored in rate-optimality theory under Gaussian equicorrelation alternatives as well as in numerical experiments for specific cases of more general alternatives.

[1]  Wang Zhou Asymptotic distribution of the largest off-diagonal entry of correlation matrices , 2007 .

[2]  T. Cai,et al.  Limiting laws of coherence of random matrices with applications to testing covariance structure and construction of compressed sensing matrices , 2011, 1102.2925.

[3]  Guangyu Mao A new test of independence for high-dimensional data , 2014 .

[4]  Maria L. Rizzo,et al.  Measuring and testing dependence by correlation of distances , 2007, 0803.4101.

[5]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[6]  W. Hoeffding A Class of Statistics with Asymptotically Normal Distribution , 1948 .

[7]  Han Liu,et al.  Distribution-Free Tests of Independence with Applications to Testing More Structures , 2014, 1410.4179.

[8]  James R. Schott,et al.  Testing for complete independence in high dimensions , 2005 .

[9]  C. F. Kossack,et al.  Rank Correlation Methods , 1949 .

[10]  Weidong Liu,et al.  Necessary and sufficient conditions for the asymptotic distribution of the largest entry of a sample correlation matrix , 2010 .

[11]  Tiefeng Jiang,et al.  The asymptotic distributions of the largest entries of sample correlation matrices , 2004, math/0406184.

[12]  R. Graham,et al.  Spearman's Footrule as a Measure of Disarray , 1977 .

[13]  R. Serfling Approximation Theorems of Mathematical Statistics , 1980 .

[14]  Wicher P. Bergsma,et al.  A consistent test of independence based on a sign covariance related to Kendall's tau , 2010, 1007.4259.

[15]  Andrew Rosalsky,et al.  On Jiang's asymptotic distribution of the largest entry of a sample correlation matrix , 2010, J. Multivar. Anal..

[16]  Q. Shao,et al.  THE ASYMPTOTIC DISTRIBUTION AND BERRY-ESSEEN BOUND OF A NEW TEST FOR INDEPENDENCE IN HIGH DIMENSION WITH AN APPLICATION TO STOCHASTIC OPTIMIZATION , 2008, 0901.2468.

[17]  P. Hall,et al.  Martingale Limit Theory and its Application. , 1984 .