Nonparametric independence testing via mutual information

We propose a test of independence of two multivariate random vectors, given a sample from the underlying population. Our approach, which we call MINT, is based on the estimation of mutual information, whose decomposition into joint and marginal entropies facilitates the use of recently-developed efficient entropy estimators derived from nearest neighbour distances. The proposed critical values, which may be obtained from simulation (in the case where one marginal is known) or resampling, guarantee that the test has nominal size, and we provide local power analyses, uniformly over classes of densities whose mutual information satisfies a lower bound. Our ideas may be extended to provide a new goodness-of-fit tests of normal linear models based on assessing the independence of our vector of covariates and an appropriately-defined notion of an error vector. The theory is supported by numerical studies on both simulated and real data.

[1]  J. E. García,et al.  A non-parametric test of independence ∗ , 2011 .

[2]  Bernhard Schölkopf,et al.  Kernel-based Conditional Independence Test and Application in Causal Discovery , 2011, UAI.

[3]  Edmond H. C. Wu,et al.  A smoothed bootstrap test for independence based on mutual information , 2009, Comput. Stat. Data Anal..

[4]  Dong Nguyen,et al.  A Kernel Independence Test for Geographical Language Variation , 2016, CL.

[5]  Pravin M. Vaidya,et al.  AnO(n logn) algorithm for the all-nearest-neighbors Problem , 1989, Discret. Comput. Geom..

[6]  Yihong Wu,et al.  Wasserstein Continuity of Entropy and Outer Bounds for Interference Channels , 2015, IEEE Transactions on Information Theory.

[7]  Mark Holmes,et al.  Tests of independence among continuous random vectors based on Cramér-von Mises functionals of the empirical copula process , 2009, J. Multivar. Anal..

[8]  J. Peixoto.  A Property of Well-Formulated Polynomial Regression Models , 1990 .

[9]  Kari Torkkola,et al.  Feature Extraction by Non-Parametric Mutual Information Maximization , 2003, J. Mach. Learn. Res..

[10]  B. Schölkopf,et al.  Kernel‐based tests for joint independence , 2016, 1603.00285.

[11]  Carsten O. Daub,et al.  The mutual information: Detecting and evaluating dependencies between variables , 2002, ECCB.

[12]  Kenji Fukumizu,et al.  Equivalence of distance-based and RKHS-based statistics in hypothesis testing , 2012, ArXiv.

[13]  Gábor J. Székely,et al.  The distance correlation t-test of independence in high dimension , 2013, J. Multivar. Anal..

[14]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[15]  J. Kinney,et al.  Equitability, mutual information, and the maximal information coefficient , 2013, Proceedings of the National Academy of Sciences.

[16]  S. Stigler.  Francis Galton's Account of the Invention of Correlation , 1989 .

[17]  X. Shao,et al.  Testing mutual independence in high dimension via distance covariance , 2016, 1609.09380.

[18]  James Bailey,et al.  Information Theoretic Measures for Clusterings Comparison: Variants, Properties, Normalization and Correction for Chance , 2010, J. Mach. Learn. Res..

[19]  M. Yuan,et al.  Independent component analysis via nonparametric maximum likelihood estimation , 2012, 1206.0457.

[20]  Rajen Dinesh Shah,et al.  Goodness‐of‐fit tests for high dimensional linear models , 2015, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[21]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[22]  Mathias Drton,et al.  Efficient computation of the Bergsma–Dassios sign covariance , 2015, Comput. Stat..

[23]  Thomas D. Sandry,et al.  Introductory Statistics With R , 2003, Technometrics.

[24]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[25]  Natalie Neumeyer,et al.  Testing independence in nonparametric regression , 2009, J. Multivar. Anal..

[26]  Le Song,et al.  Feature Selection via Dependence Maximization , 2012, J. Mach. Learn. Res..

[27]  Thomas B. Berrett,et al.  Efficient multivariate entropy estimation via $k$-nearest neighbour distances , 2016, The Annals of Statistics.

[28]  N. Meinshausen,et al.  Symmetric rank covariances: a generalized framework for nonparametric measures of dependence , 2017, Biometrika.

[29]  John W. Fisher,et al.  ICA Using Spacings Estimates of Entropy , 2003, J. Mach. Learn. Res..

[30]  Ingrid Van Keilegom,et al.  Estimating the error distribution in nonparametric multiple regression with applications to model testing , 2010, J. Multivar. Anal..

[31]  Arthur Gretton,et al.  Consistent Nonparametric Tests of Independence , 2010, J. Mach. Learn. Res..

[32]  Maria L. Rizzo,et al.  Measuring and testing dependence by correlation of distances , 2007, 0803.4101.

[33]  Eric R. Ziegel,et al.  An Introduction to Generalized Linear Models , 2002, Technometrics.

[34]  H. Joe.  Relative Entropy Measures of Multivariate Dependence , 1989 .

[35]  Thomas B. Berrett,et al.  The conditional permutation test , 2018, 1807.05405.

[36]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[37]  I. Keilegom,et al.  Tests for Independence in Nonparametric Regression , 2006 .

[38]  S. Holmes,et al.  Measuring multivariate association and beyond. , 2016, Statistics surveys.

[39]  Yanan Fan,et al.  Multivariate nonparametric test of independence , 2017, J. Multivar. Anal..

[40]  Arthur Gretton,et al.  An Adaptive Test of Independence with Analytic Kernel Embeddings , 2016, ICML.

[41]  Bodhisattva Sen,et al.  Testing independence and goodness-of-fit in linear models , 2013 .

[42]  B. Schweizer,et al.  On Nonparametric Measures of Dependence for Random Variables , 1981 .

[43]  Malka Gorfine,et al.  Consistent Distribution-Free $K$-Sample and Independence Tests for Univariate Random Variables , 2014, J. Mach. Learn. Res..

[44]  Yang Feng,et al.  A Conditional Dependence Measure with Applications to Undirected Graphical Models , 2015 .

[45]  Arthur Gretton,et al.  Large-scale kernel methods for independence testing , 2016, Statistics and Computing.

[46]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[47]  H. White,et al.  A NONPARAMETRIC HELLINGER METRIC TEST FOR CONDITIONAL INDEPENDENCE , 2008, Econometric Theory.

[48]  K. Pearson.  NOTES ON THE HISTORY OF CORRELATION , 1920 .

[49]  W. Wefelmeyer,et al.  Estimating the error distribution function in semiparametric additive regression models , 2012 .

[50]  S. Kotz,et al.  Correlation and dependence , 2001 .

[51]  Bernhard Schölkopf,et al.  Measuring Statistical Dependence with Hilbert-Schmidt Norms , 2005, ALT.

[52]  P. Reynaud-Bouret,et al.  Bootstrap and permutation tests of independence for point processes , 2014, 1406.1643.

[53]  Luc Devroye,et al.  Lectures on the Nearest Neighbor Method , 2015 .

[54]  Alison L Gibbs,et al.  On Choosing and Bounding Probability Metrics , 2002, math/0209021.

[55]  Wicher P. Bergsma,et al.  A consistent test of independence based on a sign covariance related to Kendall's tau , 2010, 1007.4259.

[56]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..