A Fast Approach for Overcomplete Sparse Decomposition Based on Smoothed $\ell ^{0}$ Norm

In this paper, a fast algorithm for overcomplete sparse decomposition, called SL0, is proposed. The algorithm is essentially a method for obtaining sparse solutions of underdetermined systems of linear equations, and its applications include underdetermined sparse component analysis (SCA), atomic decomposition on overcomplete dictionaries, compressed sensing, and decoding real field codes. Contrary to previous methods, which usually solve this problem by minimizing the l 1 norm using linear programming (LP) techniques, our algorithm tries to directly minimize the l 1 norm. It is experimentally shown that the proposed algorithm is about two to three orders of magnitude faster than the state-of-the-art interior-point LP solvers, while providing the same (or better) accuracy.

[1]  Florent Krzakala,et al.  Statistical physics-based reconstruction in compressed sensing , 2011, ArXiv.

[2]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[3]  Christian Jutten,et al.  A Fast Method for Sparse Component Analysis Based on Iterative Detection‐Estimation , 2006 .

[4]  Fabian J. Theis,et al.  Blind source separation and sparse component analysis of overcomplete mixtures , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[5]  Sacha Krstulovic,et al.  Mptk: Matching Pursuit Made Tractable , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[6]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[8]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[9]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[10]  Vahid Tarokh,et al.  A Coding Theory Approach to Noisy Compressive Sensing Using Low Density Frames , 2011, IEEE Transactions on Signal Processing.

[11]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[12]  Christian Jutten,et al.  Estimating the mixing matrix in Sparse Component Analysis (SCA) based on partial k-dimensional subspace clustering , 2008, Neurocomputing.

[13]  S. Amari,et al.  SPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES , 2003 .

[14]  R.G. Baraniuk,et al.  Compressive Sensing [Lecture Notes] , 2007, IEEE Signal Processing Magazine.

[15]  David L. Donoho,et al.  Precise Undersampling Theorems , 2010, Proceedings of the IEEE.

[16]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[17]  Barak A. Pearlmutter,et al.  Blind source separation by sparse decomposition , 2000, SPIE Defense + Commercial Sensing.

[18]  Christian Jutten,et al.  Fast Sparse Representation Based on Smoothed l0 Norm , 2007, ICA.

[19]  Robert D. Nowak,et al.  A bound optimization approach to wavelet-based image deconvolution , 2005, IEEE International Conference on Image Processing 2005.

[20]  Chih-Chun Wang,et al.  Random Sparse Linear Systems Observed Via Arbitrary Channels: A Decoupling Principle , 2007, 2007 IEEE International Symposium on Information Theory.

[21]  Michael Elad,et al.  Why Simple Shrinkage Is Still Relevant for Redundant Representations? , 2006, IEEE Transactions on Information Theory.

[22]  Rémi Gribonval,et al.  A survey of Sparse Component Analysis for blind source separation: principles, perspectives, and new challenges , 2006, ESANN.

[23]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[24]  E. Candes,et al.  11-magic : Recovery of sparse signals via convex programming , 2005 .

[25]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[26]  Andrzej Cichocki,et al.  On-Line K-PLANE Clustering Learning Algorithm for Sparse Comopnent Analysis , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[27]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[28]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[29]  M. Babaie-Zadeh,et al.  COMPLEX-VALUED SPARSE REPRESENTATION BASED ON SMOOTHED , 2007 .

[30]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[31]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[32]  Jagath C. Rajapakse,et al.  Adaptive blind signal and image processing: learning algorithms and applications: A. Cichocki, S. Amari, Wiley, New York, 2002, 586pp., ISBN 0471 60791 6 , 2002, Neurocomputing.

[33]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[34]  Christian Jutten,et al.  Complex-valued sparse representation based on smoothed ℓ0 norm , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[35]  Dongning Guo,et al.  Asymptotic Mean-Square Optimality of Belief Propagation for Sparse Linear Systems , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.

[36]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[37]  Richard G. Baraniuk,et al.  Bayesian Compressive Sensing Via Belief Propagation , 2008, IEEE Transactions on Signal Processing.

[38]  Jian Li,et al.  Computationally Efficient Sparse Bayesian Learning via Belief Propagation , 2009, IEEE Transactions on Signal Processing.

[39]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[40]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[41]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[42]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[43]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..