Discovering Potential Correlations via Hypercontractivity

Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms that we expect a measure of potential correlation to satisfy; (ii) we show that the rate of information bottleneck, i.e., the hypercontractivity coefficient, satisfies all the proposed axioms; (iii) we provide a novel estimator to estimate the hypercontractivity coefficient from samples; and (iv) we provide numerical experiments demonstrating that this proposed estimator discovers potential correlations among various indicators of WHO datasets, is robust in discovering gene interactions from gene expression time series data, and is statistically more powerful than the estimators for other correlation measures in binary hypothesis testing of canonical examples of potential correlations.

[1]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[2]  Gal Chechik,et al.  Information Bottleneck for Gaussian Variables , 2003, J. Mach. Learn. Res..

[3]  Lizhong Zheng,et al.  Linear Bounds between Contraction Coefficients for $f$-Divergences , 2015, 1510.01844.

[4]  Karen Livescu,et al.  Nonparametric Canonical Correlation Analysis , 2015, ICML.

[5]  P. Gács,et al.  Spreading of Sets in Product Spaces and Hypercontraction of the Markov Operator , 1976 .

[6]  Maria L. Rizzo,et al.  Measuring and testing dependence by correlation of distances , 2007, 0803.4101.

[7]  Nathan Linial,et al.  The influence of variables on Boolean functions , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[8]  C. B. Bell.  Mutual Information and Maximal Correlation as Measures of Dependence , 1962 .

[9]  Stefano Soatto,et al.  Information Dropout: Learning Optimal Representations Through Noisy Computation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  H. Witsenhausen.  ON SEQUENCES OF PAIRS OF DEPENDENT RANDOM VARIABLES , 1975 .

[11]  K. Pearson.  VII. Note on regression and inheritance in the case of two parents , 1895, Proceedings of the Royal Society of London.

[12]  Inderjit S. Dhillon,et al.  A Divisive Information-Theoretic Feature Clustering Algorithm for Text Classification , 2003, J. Mach. Learn. Res..

[13]  Sean C. Bendall,et al.  Conditional density-based analysis of T cell signaling in single-cell data , 2014, Science.

[14]  Venkat Anantharam,et al.  On Maximal Correlation, Hypercontractivity, and the Data Processing Inequality studied by Erkip and Cover , 2013, ArXiv.

[15]  A. Bonami.  Étude des coefficients de Fourier des fonctions de $L^p(G)$ , 1970 .

[16]  Reza Modarres,et al.  Measures of Dependence , 2011, International Encyclopedia of Statistical Science.

[17]  Edward Nelson,et al.  Construction of quantum fields from Markoff fields. , 1973 .

[18]  CHANDRA NAIR,et al.  EQUIVALENT FORMULATIONS OF HYPERCONTRACTIVITY USING INFORMATION MEASURES , 2014 .

[19]  W. Beckner.  Inequalities in Fourier analysis , 1975 .

[20]  Nitish Srivastava,et al.  Multimodal learning with deep Boltzmann machines , 2012, J. Mach. Learn. Res..

[21]  T. Sanders,et al.  Analysis of Boolean Functions , 2012, ArXiv.

[22]  Leonard Gross,et al.  Hypercontractivity and logarithmic Sobolev inequalities for the Clifford-Dirichlet form , 1975 .

[23]  Adebayo Adeloye,et al.  Replacing outliers and missing values from activated sludge data using kohonen self-organizing map , 2007 .

[24]  R. Tibshirani,et al.  Comment on "Detecting Novel Associations In Large Data Sets" by Reshef Et Al, Science Dec 16, 2011 , 2014, 1401.7645.

[25]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[26]  H. Hirschfeld.  A Connection between Correlation and Contingency , 1935, Mathematical Proceedings of the Cambridge Philosophical Society.

[27]  H. Gebelein.  Das statistische Problem der Korrelation als Variations‐ und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung , 1941 .

[28]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[29]  Elchanan Mossel,et al.  On Reverse Hypercontractivity , 2011, Geometric and Functional Analysis.

[30]  Michael Mitzenmacher,et al.  Detecting Novel Associations in Large Data Sets , 2011, Science.

[31]  Sreeram Kannan,et al.  Conditional Dependence via Shannon Capacity: Axioms, Estimators and Applications , 2016, ICML.

[32]  rian Dilvies,et al.  Hypercontractivity : A Bibliographic Review , 2022 .

[33]  Ran El-Yaniv,et al.  Distributional Word Clusters vs. Words for Text Categorization , 2003, J. Mach. Learn. Res..

[34]  Malka Gorfine,et al.  Comment on “ Detecting Novel Associations in Large Data Sets ” , 2012 .

[35]  Edward A. McBean,et al.  Statistical procedures for analysis of environmental monitoring data and risk assessment , 2000 .

[36]  Chandra Nair,et al.  AN EXTREMAL INEQUALITY RELATED TO HYPERCONTRACTIVITY OF GAUSSIAN RANDOM VARIABLES , 2014 .

[37]  Alexander A. Alemi,et al.  Deep Variational Information Bottleneck , 2017, ICLR.

[38]  F. Mosteller,et al.  Low Moments for Small Samples: A Comparative Study of Order Statistics , 1947 .