Fast Parallel Estimation of High Dimensional Information Theoretical Quantities with Low Dimensional Random Projection Ensembles

The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task. We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.

[1]  MatoušekJiří.  On variants of the JohnsonLindenstrauss lemma , 2008 .

[2]  Barnabás Póczos,et al.  Cross-Entropy Optimization for Independent Process Analysis , 2006, ICA.

[3]  Fabian J. Theis,et al.  Blind signal separation into groups of dependent signals using joint block diagonalization , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[4]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[5]  Barnabás Póczos,et al.  Undercomplete Blind Subspace Deconvolution , 2007, J. Mach. Learn. Res..

[6]  Jan Kybic.  High-dimensional mutual information estimation for image registration , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[7]  Fabian J. Theis,et al.  Uniqueness of complex and multidimensional independent component analysis , 2004, Signal Process..

[8]  Santosh S. Vempala,et al.  The Random Projection Method , 2005, DIMACS Series in Discrete Mathematics and Theoretical Computer Science.

[9]  Barnabás Póczos,et al.  Independent Subspace Analysis Using k-Nearest Neighborhood Distances , 2005, ICANN.

[10]  Dirk P. Kroese,et al.  Cross‐Entropy Method , 2011 .

[11]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[12]  Giuliano Grossi,et al.  Random Projections for Dimensionality Reduction in ICA , 2008 .

[13]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[14]  Aapo Hyvärinen,et al.  Learning Natural Image Structure with a Horizontal Product Model , 2009, ICA.

[15]  Michael I. Jordan,et al.  Beyond Independent Components: Trees and Clusters , 2003, J. Mach. Learn. Res..

[16]  Alfred O. Hero,et al.  Applications of entropic spanning graphs , 2002, IEEE Signal Process. Mag..

[17]  Santosh S. Vempala,et al.  An algorithmic theory of learning: Robust concepts and random projection , 1999, Machine Learning.

[18]  Alfred O. Hero,et al.  Image registration methods in high‐dimensional space , 2006, Int. J. Imaging Syst. Technol..

[19]  J. Matousek,et al.  On variants of the Johnson–Lindenstrauss lemma , 2008 .

[20]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[21]  Ella Bingham,et al.  Advances in independent component analysis with applications to data mining , 2003 .

[22]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[23]  Allan Kardec Barros,et al.  Independent Component Analysis and Blind Source Separation , 2007, Signal Processing.

[24]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[25]  W FisherJohn,et al.  ICA using spacings estimates of entropy , 2004 .

[26]  Jean-François Cardoso,et al.  Multidimensional independent component analysis , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[27]  Fabian J. Theis,et al.  Multidimensional independent component analysis using characteristic functions , 2005, 2005 13th European Signal Processing Conference.

[28]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[29]  John W. Fisher,et al.  ICA Using Spacings Estimates of Entropy , 2003, J. Mach. Learn. Res..

[30]  Barnabás Póczos,et al.  Independent Process Analysis Without a Priori Dimensional Information , 2007, ICA.

[31]  Erkki Oja,et al.  Artificial Neural Networks: Formal Models and Their Applications - ICANN 2005, 15th International Conference, Warsaw, Poland, September 11-15, 2005, Proceedings, Part II , 2005, International Conference on Artificial Neural Networks.