Nonlinear Independent Component Analysis for Continuous-Time Signals

We study the classical problem of recovering a multidimensional source process from observations of nonlinear mixtures of this process. Assuming statistical independence of the coordinate processes of the source, we show that this recovery is possible for many popular models of stochastic processes (up to order and monotone scaling of their coordinates) if the mixture is given by a sufficiently differentiable, invertible function. Key to our approach is the combination of tools from stochastic analysis and recent contrastive learning approaches to nonlinear ICA. This yields a scalable method with widely applicable theoretical guarantees for which our experiments indicate good performance.

[1]  Bernhard Schölkopf,et al.  Kernel Methods for Measuring Independence , 2005, J. Mach. Learn. Res..

[2]  Xiaohong Chen,et al.  Estimation of Copula-Based Semiparametric Time Series Models , 2006 .

[3]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[4]  Klaus Nordhausen,et al.  Blind Source Separation Based on Joint Diagonalization in R: The Packages JADE and BSSasymp , 2017 .

[5]  E. T. Olsen,et al.  Copulas and Markov processes , 1992 .

[6]  Patrick Kidger,et al.  Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU , 2020, ICLR.

[7]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[8]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[9]  Aapo Hyvärinen,et al.  Fast and robust fixed-point algorithms for independent component analysis , 1999, IEEE Trans. Neural Networks.

[10]  E. Lukács,et al.  A Property of the Normal Distribution , 1954 .

[11]  Aapo Hyvrinen.  Independent Component Analysis by Minimization of Mutual Information Independent Component Analysis by Minimization of Mutual Information Independent Component Analysis by Minimization of Mutual Information , 1997 .

[12]  Christian Kleiber,et al.  Multivariate distributions and the moment problem , 2013, J. Multivar. Anal..

[13]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[14]  Philippe Garat,et al.  Blind separation of mixture of independent sources through a quasi-maximum likelihood approach , 1997, IEEE Trans. Signal Process..

[15]  Kuo-Tsai Chen,et al.  Iterated path integrals , 1977 .

[16]  Takeshi Emura,et al.  R routines for performing estimation and statistical process control under copula-based time series models , 2017, Commun. Stat. Simul. Comput..

[17]  Free Lie algebras,et al.  Free Lie algebras , 2015 .

[18]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[19]  Bill Ravens,et al.  An Introduction to Copulas , 2000, Technometrics.

[20]  Motoaki Kawanabe,et al.  Kernel-Based Nonlinear Blind Source Separation , 2003, Neural Computation.

[21]  Terry Lyons,et al.  Uniqueness for the signature of a path of bounded variation and the reduced path group , 2005, math/0507536.

[22]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[23]  Robert Tibshirani,et al.  Independent Components Analysis through Product Density Estimation , 2002, NIPS.

[24]  I. Chevyrev,et al.  Signature Moments to Characterize Laws of Stochastic Processes , 2018, J. Mach. Learn. Res..

[25]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[26]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[27]  R. Bass,et al.  Review: P. Billingsley, Convergence of probability measures , 1971 .

[28]  Yoshua Bengio,et al.  Learning Independent Features with Adversarial Nets for Non-linear ICA , 2017, 1710.05050.

[29]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[30]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[31]  M. Fliess,et al.  Fonctionnelles causales non linaires et indtermines non commutatives , 1981 .

[32]  Terry Lyons,et al.  Characteristic functions of measures on geometric rough paths , 2013, 1307.3580.

[33]  P. Embrechts,et al.  Risk Management: Correlation and Dependence in Risk Management: Properties and Pitfalls , 2002 .

[34]  John M. Lee.  Introduction to Smooth Manifolds , 2002 .

[35]  Luís B. Almeida,et al.  MISEP -- Linear and Nonlinear ICA Based on Mutual Information , 2003, J. Mach. Learn. Res..

[36]  Aapo Hyvärinen,et al.  Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning , 2018, AISTATS.

[37]  Aapo Hyvärinen,et al.  Nonlinear ICA of Temporally Dependent Stationary Sources , 2017, AISTATS.

[38]  Patric Bonnier,et al.  Signature cumulants, ordered partitions, and independence of stochastic processes , 2019, 1908.06496.

[39]  Jacek M. Zurada,et al.  Nonlinear Blind Source Separation Using a Radial Basis Function Network , 2001 .

[40]  Eric Moulines,et al.  Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[41]  Aapo Hyvärinen,et al.  Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA , 2016, NIPS.

[42]  École d'été de probabilités de Saint-Flour,et al.  Differential equations driven by rough paths , 2007 .

[43]  Aapo Hyvärinen,et al.  New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit , 1997, NIPS.

[44]  M. Yuan,et al.  Independent component analysis via nonparametric maximum likelihood estimation , 2012, 1206.0457.

[45]  Tõnu Kollo,et al.  Communications in Statistics-Simulation and Computation , 2015 .

[46]  Peter K. Friz,et al.  Multidimensional Stochastic Processes as Rough Paths: Theory and Applications , 2010 .

[47]  Aapo Hyvärinen,et al.  Independent component analysis: recent advances , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.