Learning Sparse Representation Using Iterative

In this paper, we introduce the iterative subspace identification (ISI) algorithm for learning subspaces in which the data may live. Our subspace identification method differs from currently available method in its ability to infer the dimension of the subspaces from the data without prior knowledge. The learned subspaces can be combined to produce a data driven overcomplete dictionary with good sparseness and generalizability qualities, or can be directly exploited in applications where block sparseness is needed. We describe the ISI algorithm and a complementary optimization method. We demonstrate the ability of the proposed method to produce sparse representations comparable to those achieved with the K-SVD algorithm, but with less than one eighth the training time. Furthermore, the computation savings allows us to develop a shift-tolerant training procedure. We also illustrate its benefits in underdetermined blind source separation of audio, where performance is directly impacted by the sparseness of the representation.

[1]  Bruno A. Olshausen,et al.  PROBABILISTIC FRAMEWORK FOR THE ADAPTATION AND COMPARISON OF IMAGE CODES , 1999 .

[2]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[3]  Bhaskar D. Rao,et al.  FOCUSS-based dictionary learning algorithms , 2000, SPIE Optics + Photonics.

[4]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[5]  Rémi Gribonval.  Sparse decomposition of stereo signals with Matching Pursuit and application to blind separation of more than two sources from a stereo mixture , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[6]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[7]  J. Tropp.  Algorithms for simultaneous sparse approximation. Part II: Convex relaxation , 2006, Signal Process..

[8]  Xiaoming Huo,et al.  Beamlets and Multiscale Image Analysis , 2002 .

[9]  Ahmed H. Tewfik,et al.  Overcoming measurement time variability in brain machine interface , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[11]  Rémi Gribonval,et al.  Learning unions of orthonormal bases with thresholded singular value decomposition , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[12]  Ahmed H. Tewfik,et al.  Two Improved Sparse Decomposition Methods for Blind Source Separation , 2007, ICA.

[13]  Ahmed H. Tewfik,et al.  Blind source separation using monochannel overcomplete dictionaries , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[15]  Mike E. Davies,et al.  Dictionary Learning for Sparse Approximations With the Majorization Method , 2009, IEEE Transactions on Signal Processing.

[16]  Joseph F. Murray,et al.  An improved FOCUSS-based learning algorithm for solving sparse linear inverse problems , 2001, Conference Record of Thirty-Fifth Asilomar Conference on Signals, Systems and Computers (Cat.No.01CH37256).

[17]  Kjersti Engan,et al.  Multi-frame compression: theory and design , 2000, Signal Process..

[18]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[19]  Dan Wang,et al.  Sparse representation of deformable 3D organs , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[20]  Paul S. Bradley,et al.  k-Plane Clustering , 2000, J. Glob. Optim..

[21]  Dan Wang,et al.  In vivo tracking of 3D organs using spherical harmonics and subspace clustering , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[22]  Ahmed H. Tewfik,et al.  A sparse solution to the bounded subset selection problem: a network flow model approach , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[23]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[24]  Pierre Vandergheynst,et al.  Shift-invariant dictionary learning for sparse representations: Extending K-SVD , 2008, 2008 16th European Signal Processing Conference.

[25]  Rémi Gribonval,et al.  Performance measurement in blind audio source separation , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[26]  Yonina C. Eldar,et al.  Block-sparsity: Coherence and efficient recovery , 2008, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  P. Tseng.  Nearest q-Flat to m Points , 2000 .

[28]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[29]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[30]  D. Donoho,et al.  Atomic Decomposition by Basis Pursuit , 2001 .

[31]  David L. Donoho,et al.  Digital curvelet transform: strategy, implementation, and experiments , 2000, SPIE Defense + Commercial Sensing.

[32]  Scott Rickard,et al.  Blind separation of speech mixtures via time-frequency masking , 2004, IEEE Transactions on Signal Processing.

[33]  Pierre Vandergheynst,et al.  MoTIF: An Efficient Algorithm for Learning Translation Invariant Dictionaries , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[34]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[35]  Simon J. Godsill,et al.  A Bayesian Approach for Blind Separation of Sparse Sources , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[36]  Sacha Krstulovic,et al.  Under-Determined Source Separation: Comparison of Two Approaches Based on Sparse Decompositions , 2006, ICA.

[37]  T. Blumensath,et al.  On the Difference Between Orthogonal Matching Pursuit and Orthogonal Least Squares , 2007 .

[38]  Mike E. Davies,et al.  Sparse and shift-Invariant representations of music , 2006, IEEE Transactions on Audio, Speech, and Language Processing.