Scale Invariant Conditional Dependence Measures

In this paper we develop new dependence and conditional dependence measures and provide their estimators. An attractive property of these measures and estimators is that they are invariant to any monotone increasing transformations of the random variables, which is important in many applications including feature selection. Under certain conditions we show the consistency of these estimators, derive upper bounds on their convergence rates, and show that the estimators do not suffer from the curse of dimensionality. However, when the conditions are less restrictive, we derive a lower bound which proves that in the worst case the convergence can be arbitrarily slow similarly to some other estimators. Numerical illustrations demonstrate the applicability of our method.

[1]  R. Fortet,et al.  Convergence de la répartition empirique vers la répartition théorique , 1953 .

[2]  A. Rényi.  On measures of dependence , 1959 .

[3]  A. Rényi.  On Measures of Entropy and Information , 1961 .

[4]  Edwin Hewitt,et al.  Real and Abstract Analysis: A Modern Treatment of the Theory of Functions of a Real Variable , 1965 .

[5]  C. Baker.  Joint measures and cross-covariance operators , 1973 .

[6]  B. Schweizer,et al.  On Nonparametric Measures of Dependence for Random Variables , 1981 .

[7]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[8]  P. Bickel,et al.  Achieving Information Bounds in Non and Semiparametric Models , 1990 .

[9]  Alexander J. Smola,et al.  The kernel mutual information , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[10]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003 .

[11]  Michael I. Jordan,et al.  Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces , 2004, J. Mach. Learn. Res..

[12]  Kenji Fukumizu,et al.  Statistical Convergence of Kernel CCA , 2005, NIPS.

[13]  Bernhard Schölkopf,et al.  Measuring Statistical Dependence with Hilbert-Schmidt Norms , 2005, ALT.

[14]  Hans-Peter Kriegel,et al.  Integrating structured biological data by Kernel Maximum Mean Discrepancy , 2006, ISMB.

[15]  Bernhard Schölkopf,et al.  Kernel Measures of Conditional Dependence , 2007, NIPS.

[16]  Maria L. Rizzo,et al.  Measuring and testing dependence by correlation of distances , 2007, 0803.4101.

[17]  Barnabás Póczos,et al.  Nonparametric Estimation of Conditional Information and Divergences , 2012, AISTATS.

[18]  Guy Lever,et al.  Conditional mean embeddings as regressors , 2012, ICML.

[19]  Barnabás Póczos,et al.  Copula-based Kernel Dependency Measures , 2012, ICML.