Blind Source Separation: the Sparsity Revolution

Over the last few years, the development of multi-channel sensors motivated interest in methods for the coherent processing of multivariate data. Some specific issues have already been addressed as testified by the wide literature on the so-called blind source separation (BSS) problem. In this context, as clearly emphasized by previous work, it is fundamental that the sources to be retrieved present some quantitatively measurable diversity. Recently, sparsity and morphological diversity have emerged as a novel and effective source of diversity for BSS. We give here some essential insights into the use of sparsity in source separation and we outline the essential role of morphological diversity as being a source of diversity or contrast between the sources. This paper overviews a sparsity-based BSS method coined Generalized Morphological Component Analysis (GMCA) that takes advantages of both morphological diversity and sparsity, using recent sparse overcomplete or redundant signal representations. GMCA is a fast and efficient blind source separation method. In remote sensing applications, the specificity of hyperspectral data should be accounted for. We extend the proposed GMCA framework to deal with hyperspectral data. In a general framework, GMCA provides a basis for multivariate data analysis in the scope of a wide range of classical multivariate data restorate. Numerical results are given in color image denoising and inpainting. Finally, GMCA is applied to the simulated ESA/Planck data. It is shown to give effective astrophysical component separation.

[1]  Barak A. Pearlmutter,et al.  Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA , 1996, NIPS.

[2]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[3]  Mohamed-Jalal Fadili,et al.  Sparsity and Morphological Diversity in Blind Source Separation , 2007, IEEE Transactions on Image Processing.

[4]  Jean-Luc Starck,et al.  Enhanced Source Separation by Morphological Component Analysis , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[5]  Michael Elad,et al.  Submitted to Ieee Transactions on Image Processing Image Decomposition via the Combination of Sparse Representations and a Variational Approach , 2022 .

[6]  Arkadi Nemirovski,et al.  On sparse representation in pairs of bases , 2003, IEEE Trans. Inf. Theory.

[7]  Minh N. Do,et al.  Ieee Transactions on Image Processing the Contourlet Transform: an Efficient Directional Multiresolution Image Representation , 2022 .

[8]  E. Candès,et al.  Ridgelets: a key to higher-dimensional intermittency? , 1999, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[9]  A. J. Bell,et al.  A Unifying Information-Theoretic Framework for Independent Component Analysis , 2000 .

[10]  Naoki Saito,et al.  Sparsity vs. Statistical Independence in Adaptive Signal Representations: A Case Study of the Spike Process , 2001, math/0104083.

[11]  Edward J. Wollack,et al.  First-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Preliminary Maps and Basic Results , 2003, astro-ph/0302207.

[12]  Shun-ichi Amari,et al.  Superefficiency in blind source separation , 1999, IEEE Trans. Signal Process..

[13]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[14]  S. Mallat A wavelet tour of signal processing , 1998 .

[15]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[16]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[17]  T. Ens,et al.  Blind signal separation : statistical principles , 1998 .

[18]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[19]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[20]  Morten Nielsen,et al.  Beyond sparsity: Recovering structured representations by ${\ell}^1$ minimization and greedy algorithms , 2007, Adv. Comput. Math..

[21]  Mark D. Plumbley Recovery of Sparse Representations by Polytope Faces Pursuit , 2006, ICA.

[22]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[23]  Zbynek Koldovský,et al.  Methods of Fair Comparison of Performance of Linear ICA Techniques in Presence of Additive Noise , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[24]  Massimo Fornasier,et al.  Recovery Algorithms for Vector-Valued Data with Joint Sparsity Constraints , 2008, SIAM J. Numer. Anal..

[25]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[26]  M. Zibulevsky BLIND SOURCE SEPARATION WITH RELATIVE NEWTON METHOD , 2003 .

[27]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[28]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[29]  Jean-Jacques Fuchs Recovery Conditions of Sparse Representations in the Presence of Noise. , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[30]  J. Cardoso Infomax and maximum likelihood for blind source separation , 1997, IEEE Signal Processing Letters.

[31]  Dinh Tuan Pham,et al.  Separation of a mixture of independent sources through a maximum likelihood approach , 1992 .

[32]  Yuanqing Li,et al.  Probability estimation for recoverability analysis of blind source separation based on sparse representation , 2006, IEEE Transactions on Information Theory.

[33]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[34]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[35]  L. Demanet,et al.  Wave atoms and sparsity of oscillatory patterns , 2007 .

[36]  Gabriel Peyré,et al.  Best Basis Compressed Sensing , 2007, IEEE Transactions on Signal Processing.

[37]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[38]  Pierre Vandergheynst,et al.  On the exponential convergence of matching pursuits in quasi-incoherent dictionaries , 2006, IEEE Transactions on Information Theory.

[39]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[40]  D. Donoho,et al.  Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA) , 2005 .

[41]  David J. Field,et al.  Wavelets, vision and the statistics of natural scenes , 1999, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[42]  Shun-ichi Amari,et al.  Blind source separation-semiparametric statistical approach , 1997, IEEE Trans. Signal Process..

[43]  Emmanuel J. Candès,et al.  The curvelet transform for image denoising , 2002, IEEE Trans. Image Process..

[44]  J. Nadal Non linear neurons in the low noise limit : a factorial code maximizes information transferJean , 1994 .

[45]  Barak A. Pearlmutter,et al.  Blind source separation by sparse decomposition , 2000, SPIE Defense + Commercial Sensing.

[46]  J. Nadal,et al.  Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer Network 5 , 1994 .

[47]  P. Tseng,et al.  Block Coordinate Relaxation Methods for Nonparametric Wavelet Denoising , 2000 .

[48]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[49]  D. Donoho Unconditional Bases Are Optimal Bases for Data Compression and for Statistical Estimation , 1993 .

[50]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[51]  Michael Elad,et al.  Sparse Representation for Color Image Restoration , 2008, IEEE Transactions on Image Processing.

[52]  P. Laguna,et al.  Signal Processing , 2002, Yearbook of Medical Informatics.

[53]  Emmanuel Vincent,et al.  Complex Nonconvex l p Norm Minimization for Underdetermined Source Separation , 2007, ICA.

[54]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[55]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[56]  J. Tropp,et al.  SIGNAL RECOVERY FROM PARTIAL INFORMATION VIA ORTHOGONAL MATCHING PURSUIT , 2005 .

[57]  Jean-Luc Starck,et al.  Morphological diversity and sparsity: new insights into multivariate data analysis , 2007, SPIE Optical Engineering + Applications.

[58]  Mohamed-Jalal Fadili,et al.  Inpainting and Zooming Using Sparse Representations , 2009, Comput. J..

[59]  J. Tropp JUST RELAX: CONVEX PROGRAMMING METHODS FOR SUBSET SELECTION AND SPARSE APPROXIMATION , 2004 .

[60]  M. Davies,et al.  Identifiability issues in noisy ICA , 2004, IEEE Signal Processing Letters.

[61]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[62]  Mohamed-Jalal Fadili,et al.  The Undecimated Wavelet Decomposition and its Reconstruction , 2007, IEEE Transactions on Image Processing.

[63]  Özgür Yilmaz,et al.  Blind separation of disjoint orthogonal signals: demixing N sources from 2 mixtures , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[64]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[65]  Ali Mohammad-Djafari,et al.  Hidden Markov models for wavelet-based blind source separation , 2006, IEEE Transactions on Image Processing.

[66]  Mohamed-Jalal Fadili,et al.  Morphological Diversity and Sparsity for Multichannel Data Restoration , 2009, Journal of Mathematical Imaging and Vision.

[67]  Enst Dependence, correlation and Gaussianity in independent component analysis , 2003 .

[68]  Erkki Oja,et al.  Efficient Variant of Algorithm FastICA for Independent Component Analysis Attaining the CramÉr-Rao Lower Bound , 2006, IEEE Transactions on Neural Networks.

[69]  Bruno A. Olshausen,et al.  Learning Sparse Multiscale Image Representations , 2002, NIPS.

[70]  Richard Gispert,et al.  Foregrounds and CMB experiments: I. Semi-analytical estimates of contamination , 1999 .

[71]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[72]  Dmitry M. Malioutov,et al.  Homotopy continuation for sparse signal representation , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[73]  Jean-Franois Cardoso High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[74]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[75]  Michael Elad,et al.  Morphological diversity and source separation , 2006, IEEE Signal Processing Letters.

[76]  Gabriel Peyré,et al.  Texture Synthesis and Modification with a Patch-Valued Wavelet Transform , 2007, SSVM.

[77]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[78]  Jean-Luc Starck,et al.  Blind Component Separation in Wavelet Space: Application to CMB Analysis , 2005, EURASIP J. Adv. Signal Process..

[79]  Jean-François Cardoso,et al.  Dependence, Correlation and Gaussianity in Independent Component Analysis , 2003, J. Mach. Learn. Res..

[80]  Michael Elad,et al.  A generalized uncertainty principle and sparse representation in pairs of bases , 2002, IEEE Trans. Inf. Theory.

[81]  P. Tichavský,et al.  Efficient variant of algorithm fastica for independent component analysis attaining the cramer-RAO lower bound , 2005, IEEE/SP 13th Workshop on Statistical Signal Processing, 2005.

[82]  Radu V. Balan,et al.  Estimator for Number of Sources Using Minimum Description Length Criterion for Blind Sparse Source Mixtures , 2007, ICA.

[83]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[84]  Eric Moulines,et al.  A blind source separation technique using second-order statistics , 1997, IEEE Trans. Signal Process..

[85]  Jie Chen,et al.  Sparse representations for multiple measurement vectors (MMV) in an over-complete dictionary , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[86]  Martin Vetterli,et al.  Wavelets, approximation, and compression , 2001, IEEE Signal Process. Mag..

[87]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[88]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[89]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[90]  Mohamed-Jalal Fadili,et al.  Morphological Component Analysis: An Adaptive Thresholding Strategy , 2007, IEEE Transactions on Image Processing.

[91]  Fabian J. Theis,et al.  Sparse component analysis and blind source separation of underdetermined mixtures , 2005, IEEE Transactions on Neural Networks.

[92]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[93]  Rémi Gribonval,et al.  Beyond sparsity: recovering structured representations by l¹ minimization and greedy algorithms. - Application to the analysis of sparse underdetermined ICA - , 2005 .

[94]  Spergel,et al.  Cosmological-parameter determination with microwave background maps. , 1996, Physical review. D, Particles and fields.

[95]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[96]  Yehoshua Y. Zeevi,et al.  Sparse ICA for blind separation of transmitted and reflected images , 2005, Int. J. Imaging Syst. Technol..

[97]  Martin Vetterli,et al.  Data Compression and Harmonic Analysis , 1998, IEEE Trans. Inf. Theory.

[98]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[99]  Stéphane Mallat,et al.  Sparse geometric image representations with bandelets , 2005, IEEE Transactions on Image Processing.

[100]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[101]  Laurent Demanet,et al.  Fast Discrete Curvelet Transforms , 2006, Multiscale Model. Simul..

[102]  Jean-Francois Cardoso,et al.  THE THREE EASY ROUTES TO INDEPENDENT COMPONENT ANALYSIS; CONTRASTS AND GEOMETRY , 2001 .

[103]  D. Donoho,et al.  Redundant Multiscale Transforms and Their Application for Morphological Component Separation , 2004 .