Improved M-FOCUSS Algorithm With Overlapping Blocks for Locally Smooth Sparse Signals

The focal underdetermined system solver (FOCUSS) algorithm has already found many applications in signal processing and data analysis, whereas the regularized M-FOCUSS algorithm has been recently proposed by Cotter for finding sparse solutions to an underdetermined system of linear equations with multiple measurement vectors. In this paper, we propose three modifications to the M-FOCUSS algorithm to make it more efficient for sparse and locally smooth solutions. First, motivated by the simultaneously autoregressive (SAR) model, we incorporate an additional weighting (smoothing) matrix into the Tikhonov regularization term. Next, the entire set of measurement vectors is divided into blocks, and the solution is updated sequentially, based on the overlapping of data blocks. The last modification is based on an alternating minimization technique to provide data-driven (simultaneous) estimation of the regularization parameter with the generalized cross-validation (GCV) approach. Finally, the simulation results demonstrating the benefits of the proposed modifications support the analysis.

[1]  J. Tropp.  Algorithms for simultaneous sparse approximation. Part II: Convex relaxation , 2006, Signal Process..

[2]  Rémi Gribonval,et al.  Learning unions of orthonormal bases with thresholded singular value decomposition , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[3]  B. Rao,et al.  ℓâ‚€-norm Minimization for Basis Selection , 2004, NIPS 2004.

[4]  Michael A. Malcolm,et al.  Computer methods for mathematical computations , 1977 .

[5]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[6]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[7]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[8]  Soo Yeol Lee,et al.  Reconstruction of Continuous and Focalized Brain Functional Source Images From Electroencephalography , 2006, IEEE Transactions on Magnetics.

[9]  Christian Jutten,et al.  Multichannel ECG and Noise Modeling: Application to Maternal and Fetal ECG Signals , 2007 .

[10]  Richard M. Leahy,et al.  MEG-based imaging of focal neuronal current sources , 1996, IEEE Transactions on Medical Imaging.

[11]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[12]  D. Donoho,et al.  Uncertainty principles and signal recovery , 1989 .

[13]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[14]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[15]  K. Kreutz-Delgado,et al.  Deriving algorithms for computing sparse solutions to linear inverse problems , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[16]  S. O. Aase,et al.  Finding sparse representation of quantized frame coefficients using 0-1 integer programming , 2001, ISPA 2001. Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis. In conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat..

[17]  Joseph F. Murray,et al.  An improved FOCUSS-based learning algorithm for solving sparse linear inverse problems , 2001, Conference Record of Thirty-Fifth Asilomar Conference on Signals, Systems and Computers (Cat.No.01CH37256).

[18]  Rémi Gribonval.  Sparse decomposition of stereo signals with Matching Pursuit and application to blind separation of more than two sources from a stereo mixture , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[19]  Karl Skretting,et al.  Sparse Signal Representation using Overlapping Frames , 2002 .

[20]  Timo Similä.  Majorize-Minimize Algorithm for Multiresponse Sparse Regression , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[21]  Nikolas P. Galatsanos,et al.  Hierarchical Bayesian image restoration from partially known blurs , 2000, IEEE Trans. Image Process..

[22]  B. Rao.  Analysis and extensions of the FOCUSS algorithm , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[23]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[24]  Joseph F. Murray,et al.  Learning Sparse Overcomplete Codes for Images , 2006, J. VLSI Signal Process..

[25]  Michael Elad,et al.  Morphological diversity and source separation , 2006, IEEE Signal Processing Letters.

[26]  Andrzej Cichocki,et al.  Blind source separation: new tools for extraction of source signals and denoising , 2005, SPIE Defense + Commercial Sensing.

[27]  D. Donoho.  Superresolution via sparsity constraints , 1992 .

[28]  Mohamed-Jalal Fadili,et al.  Sparsity and Morphological Diversity in Blind Source Separation , 2007, IEEE Transactions on Image Processing.

[29]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[30]  Aggelos K. Katsaggelos,et al.  SPECT image reconstruction using compound models , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[31]  Aggelos K. Katsaggelos,et al.  Bayesian and regularization methods for hyperparameter estimation in image restoration , 1999, IEEE Trans. Image Process..

[32]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[33]  S. Amari,et al.  SPARSE COMPONENT ANALYSIS FOR BLIND SOURCE SEPARATION WITH LESS SENSORS THAN SOURCES , 2003 .

[34]  Barak A. Pearlmutter,et al.  Blind separation of sources with sparse representations in a given signal dictionary , 2000 .

[35]  Bhaskar D. Rao,et al.  Subset selection in noise based on diversity measure minimization , 2003, IEEE Trans. Signal Process..

[36]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[37]  Kjersti Engan,et al.  Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation , 2007, Digit. Signal Process..

[38]  Jie Chen,et al.  Theoretical Results on Sparse Representations of Multiple-Measurement Vectors , 2006, IEEE Transactions on Signal Processing.

[39]  Joseph F. Murray,et al.  An Improved FOCUSS-Based Learning Algorithm for Solving Blind Sparse Linear Inverse Problems , 2001 .

[40]  Brian D. Jeffs.  Sparse inverse solution methods for signal and image processing applications , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[41]  Dezhong Yao,et al.  Lp Norm Iterative Sparse Solution for EEG Source Localization , 2007, IEEE Transactions on Biomedical Engineering.

[42]  Fusheng Yang,et al.  Standardized shrinking LORETA-FOCUSS (SSLOFO): a new algorithm for spatio-temporal EEG source reconstruction , 2005, IEEE Transactions on Biomedical Engineering.

[43]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[44]  Steven W. Zucker,et al.  Greedy Basis Pursuit , 2007, IEEE Transactions on Signal Processing.

[45]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[46]  G. Wahba.  Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy , 1977 .

[47]  J. E. Moran,et al.  Multi-Resolution FOCUSS: A Source Imaging Technique Applied to MEG Data , 2005, Brain Topography.

[48]  L. Karlovitz,et al.  Construction of nearest points in the Lp, p even, and L∞ norms. I , 1970 .

[49]  Per Christian Hansen,et al.  REGULARIZATION TOOLS: A Matlab package for analysis and solution of discrete ill-posed problems , 1994, Numerical Algorithms.

[50]  Bhaskar D. Rao,et al.  Diversity measure minimization based method for computing sparse solutions to linear inverse problems with multiple measurement vectors , 2004, ICASSP.

[51]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[52]  Brian D. Ripley,et al.  Spatial Statistics: Ripley/Spatial Statistics , 2005 .

[53]  Karl Skretting,et al.  General design algorithm for sparse frame expansions , 2006, Signal Process..

[54]  B. Logan,et al.  Signal recovery and the large sieve , 1992 .

[55]  Bhaskar D. Rao,et al.  Sparse solutions to linear inverse problems with multiple measurement vectors , 2005, IEEE Transactions on Signal Processing.

[56]  Stefano Alliney,et al.  Digital filters as absolute norm regularizers , 1992, IEEE Trans. Signal Process..

[57]  F. Grund.  Forsythe, G. E. / Malcolm, M. A. / Moler, C. B., Computer Methods for Mathematical Computations. Englewood Cliffs, New Jersey 07632. Prentice Hall, Inc., 1977. XI, 259 S , 1979 .

[58]  Stefano Alliney,et al.  An algorithm for the minimization of mixed l1 and l2 norms with application to Bayesian estimation , 1994, IEEE Trans. Signal Process..

[59]  Bhaskar D. Rao,et al.  L_0-norm Minimization for Basis Selection , 2004, NIPS.

[60]  Ping Cheng,et al.  ISAR Imaging Based on Sparse Signal Representation with Multiple Measurement Vectors , 2006, 2006 CIE International Conference on Radar.

[61]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[62]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[63]  P. Whittle.  ON STATIONARY PROCESSES IN THE PLANE , 1954 .

[64]  Julian Besag,et al.  Digital Image Processing: Towards Bayesian image analysis , 1989 .

[65]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[66]  Daniel W. C. Ho,et al.  Underdetermined blind source separation based on sparse representation , 2006, IEEE Transactions on Signal Processing.

[67]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[68]  Donald L. Duttweiler,et al.  Proportionate normalized least-mean-squares adaptation in echo cancelers , 2000, IEEE Trans. Speech Audio Process..

[69]  Mohamed-Jalal Fadili,et al.  Morphological Diversity and Sparsity in Blind Source Separation , 2007, ICA.

[70]  Bhaskar D. Rao,et al.  A recursive weighted minimum norm algorithm: Analysis and applications , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[71]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Per Christian Hansen,et al.  Rank-Deficient and Discrete Ill-Posed Problems , 1996 .

[73]  Mike E. Davies,et al.  Sparse audio representations using the MCLT , 2006, Signal Process..

[74]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[75]  Mineichi Kudo,et al.  Performance analysis of minimum /spl lscr//sub 1/-norm solutions for underdetermined source separation , 2004, IEEE Transactions on Signal Processing.

[76]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[77]  Guang-Ming Zhang,et al.  Effect of sparse basis selection on ultrasonic signal representation. , 2006, Ultrasonics.

[78]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[79]  Christian Jutten,et al.  Multichannel ECG and Noise Modeling: Application to Maternal and Fetal ECG Signals , 2007, EURASIP J. Adv. Signal Process..

[80]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[81]  Jean-Luc Starck,et al.  Morphological diversity and sparsity: new insights into multivariate data analysis , 2007, SPIE Optical Engineering + Applications.

[82]  Aggelos K. Katsaggelos,et al.  SPECT Image Reconstruction Using Compound Prior Models , 2002, Int. J. Pattern Recognit. Artif. Intell..

[83]  Joel A. Tropp,et al.  Sparse Approximation Via Iterative Thresholding , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[84]  Mike E. Davies,et al.  Sparsifying subband decompositions , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[85]  E. Bullmore,et al.  Penalized partially linear models using sparse representations with an application to fMRI time series , 2005, IEEE Transactions on Signal Processing.

[86]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[87]  Brian D. Jeffs,et al.  Restoration of blurred star field images by maximally sparse optimization , 1993, IEEE Trans. Image Process..

[88]  Julian Besag,et al.  Towards Bayesian image analysis , 1993 .

[89]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[90]  Kwang Suk Park,et al.  Regularized FOCUSS algorithm for EEG/MEG source imaging , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[91]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[92]  Bhaskar D. Rao,et al.  FOCUSS-based dictionary learning algorithms , 2000, SPIE Optics + Photonics.

[93]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[94]  P. Green.  Bayesian reconstructions from emission tomography data using a modified EM algorithm. , 1990, IEEE transactions on medical imaging.