Identifiability-Guaranteed Simplex-Structured Post-Nonlinear Mixture Learning via Autoencoder

This work focuses on the problem of unraveling nonlinearly mixed latent components in an unsupervised manner. The latent components are assumed to reside in the probability simplex, and are transformed by an unknown post-nonlinear mixing system. This problem finds various applications in signal and data analytics, e.g., nonlinear hyperspectral unmixing, image embedding, and nonlinear clustering. Linear mixture learning problems are already ill-posed, as identifiability of the target latent components is hard to establish in general. With unknown nonlinearity involved, the problem is even more challenging. Prior work offered a function equation-based formulation for provable latent component identification. However, the identifiability conditions are somewhat stringent and unrealistic. In addition, the identifiability analysis is based on the infinite sample (i.e., population) case, while the understanding for practical finite sample cases has been elusive. Moreover, the algorithm in the prior work trades model expressiveness with computational convenience, which often hinders the learning performance. Our contribution is threefold. First, new identifiability conditions are derived under largely relaxed assumptions. Second, comprehensive sample complexity results are presented—which are the first of the kind. Third, a constrained autoencoder-based algorithmic framework is proposed for implementation, which effectively circumvents the challenges in the existing algorithm. Synthetic and real experiments corroborate our theoretical analyses.

[1]  Aapo Hyvärinen,et al.  Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA , 2016, NIPS.

[2]  Alfred O. Hero,et al.  Nonlinear Unmixing of Hyperspectral Images: Models and Algorithms , 2013, IEEE Signal Processing Magazine.

[3]  Motoaki Kawanabe,et al.  Blind Separation of Post-nonlinear Mixtures using Linearizing Transformations and Temporal Decorrelation , 2003, J. Mach. Learn. Res..

[4]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[5]  Yongchao Zhao,et al.  Improving the Accuracy of the Water Surface Cover Type in the 30 m FROM-GLC Product , 2015, Remote. Sens..

[6]  Andreas Ziehe,et al.  Artifact Reduction in Magnetoneurography Based on Time-Delayed Second Order Correlations , 1998 .

[7]  Vince D. Calhoun,et al.  Joint Blind Source Separation by Multiset Canonical Correlation Analysis , 2009, IEEE Transactions on Signal Processing.

[8]  Paul D. Gader,et al.  A Signal Processing Perspective on Hyperspectral Unmixing , 2014 .

[9]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[10]  Christian Jutten,et al.  Identifiability of post-nonlinear mixtures , 2005, IEEE Signal Processing Letters.

[11]  Christian Jutten,et al.  Source separation based processing for integrated Hall sensor arrays , 2002 .

[12]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[13]  Ioannis Patras,et al.  Nonlinear Independent Component Analysis for EEG-Based Brain-Computer Interface Systems , 2012 .

[14]  Ka Yee Yeung,et al.  Details of the Adjusted Rand index and Clustering algorithms Supplement to the paper “ An empirical study on Principal Component Analysis for clustering gene expression data ” ( to appear in Bioinformatics ) , 2001 .

[15]  Peter L. Bartlett,et al.  Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..

[16]  Bo Yang,et al.  Learning Nonlinear Mixtures: Identifiability and Algorithm , 2019, IEEE Transactions on Signal Processing.

[17]  L. Ronkin Liouville's theorems for functions holomorphic on the zero set of a polynomial , 1979 .

[18]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[19]  Mingyi Hong,et al.  Penalty Dual Decomposition Method For Nonsmooth Nonconvex Optimization , 2017, ArXiv.

[20]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[21]  Aapo Hyvärinen,et al.  Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning , 2018, AISTATS.

[22]  Nicolas Gillis,et al.  Robust near-separable nonnegative matrix factorization using linear optimization , 2013, J. Mach. Learn. Res..

[23]  José M. Bioucas-Dias,et al.  Does independent component analysis play a role in unmixing hyperspectral data? , 2003, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[25]  Christian Jutten,et al.  A second-order statistics method for blind source separation in post-nonlinear mixtures , 2019, Signal Process..

[26]  S. Ustin,et al.  Application of multiple endmember spectral mixture analysis (MESMA) to AVIRIS imagery for coastal salt marsh mapping: a case study in China Camp, CA, USA , 2005 .

[27]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[28]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .

[29]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[30]  Erkki Oja,et al.  The nonlinear PCA learning rule in independent component analysis , 1997, Neurocomputing.

[31]  Xiao Fu,et al.  Nonlinear Multiview Analysis: Identifiability and Neural Network-Assisted Implementation , 2020, IEEE Transactions on Signal Processing.

[32]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[33]  Jean-Yves Tourneret,et al.  Unsupervised Post-Nonlinear Unmixing of Hyperspectral Images Using a Hamiltonian Monte Carlo Algorithm , 2014, IEEE Transactions on Image Processing.

[34]  Sergio Cruces,et al.  Bounded Component Analysis of Linear Mixtures: A Criterion of Minimum Convex Perimeter , 2010, IEEE Transactions on Signal Processing.

[35]  Wing-Kin Ma,et al.  Nonnegative Matrix Factorization for Signal and Data Analytics: Identifiability, Algorithms, and Applications , 2018, IEEE Signal Processing Magazine.

[36]  Kejun Huang,et al.  Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms , 2019, NeurIPS.

[37]  Farid Oveisi EEG signal classification using nonlinear independent component analysis , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[38]  Xiao Fu,et al.  Nonlinear Dependent Component Analysis: Identifiability and Algorithm , 2021, 2020 28th European Signal Processing Conference (EUSIPCO).

[39]  Chong-Yung Chi,et al.  A Convex Analysis-Based Minimum-Volume Enclosing Simplex Algorithm for Hyperspectral Unmixing , 2009, IEEE Transactions on Signal Processing.

[40]  Aapo Hyvärinen,et al.  Nonlinear independent component analysis: Existence and uniqueness results , 1999, Neural Networks.

[41]  Christian Jutten,et al.  ISFET Source Separation: Foundations and Techniques , 2006 .

[42]  Bo Yang,et al.  Robust Volume Minimization-Based Matrix Factorization for Remote Sensing and Document Clustering , 2016, IEEE Transactions on Signal Processing.

[43]  Bernhard Schölkopf,et al.  The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA , 2019, UAI.

[44]  Min Zhao,et al.  Nonlinear Unmixing of Hyperspectral Data via Deep Autoencoder Networks , 2019, IEEE Geoscience and Remote Sensing Letters.

[45]  Zhaoshui He,et al.  Minimum-Volume-Constrained Nonnegative Matrix Factorization: Enhanced Ability of Learning Parts , 2011, IEEE Transactions on Neural Networks.

[46]  Nikos D. Sidiropoulos,et al.  Blind Separation of Quasi-Stationary Sources: Exploiting Convex Geometry in Covariance Domain , 2015, IEEE Transactions on Signal Processing.

[47]  Xiao Fu,et al.  Detecting Overlapping and Correlated Communities without Pure Nodes: Identifiability and Algorithm , 2019, ICML.

[48]  Yannick Deville,et al.  From separability/identifiability properties of bilinear and linear-quadratic mixture matrix factorization to factorization algorithms , 2019, Digit. Signal Process..

[49]  Lawrence E. Larson,et al.  Radio frequency integrated circuit technology for low-power wireless communications , 1998, IEEE Wirel. Commun..