A Local Characterization for Wyner Common Information

While the Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and the Wyner common information share similar information processing purposes of extracting common knowledge structures between random variables, the relationships between these approaches are generally unclear. In this paper, we demonstrate such relationships by considering the Wyner common information in the weakly dependent regime, called ϵ-common information. We show that the HGR maximal correlation functions coincide with the relative likelihood functions of estimating the auxiliary random variables in ϵ-common information, which establishes the fundamental connections these approaches. Moreover, we extend the ϵ-common information to multiple random variables, and derive a novel algorithm for extracting feature functions of data variables regarding their common information. Our approach is validated by the MNIST problem, and can potentially be useful in multi-modal data analyses.

[1]  Shao-Lun Huang,et al.  An information-theoretic approach to unsupervised feature selection for high-dimensional data , 2017, 2017 IEEE Information Theory Workshop (ITW).

[2]  H. Hirschfeld.  A Connection between Correlation and Contingency , 1935, Mathematical Proceedings of the Cambridge Philosophical Society.

[3]  H. Gebelein.  Das statistische Problem der Korrelation als Variations‐ und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung , 1941 .

[4]  P. Gill,et al.  Chapter III Constrained nonlinear programming , 1989 .

[5]  Reza Modarres,et al.  Measures of Dependence , 2011, International Encyclopedia of Statistical Science.

[6]  Xiangxiang Xu,et al.  An Information Theoretic Interpretation to Deep Neural Networks , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[7]  Wei Liu,et al.  The common information of N dependent random variables , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[8]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[9]  Shao-Lun Huang,et al.  On Universal Features for High-Dimensional Learning and Inference , 2019, ArXiv.