Search-based methods for the sparse signal recovery problem in compressed

The sparse signal recovery, which appears not only in compressed sensing but also in other related problems such as sparse overcomplete representations, denoising, sparse learning, etc. has drawn a large attraction in the last decade. The literature contains a vast number of recovery methods, which have been analysed in theoretical and empirical aspects. This dissertation presents novel search-based sparse signal recovery methods. First, we discuss theoretical analysis of the orthogonal matching pursuit algorithm with more iterations than the number of nonzero elements of the underlying sparse signal. Second, best-first tree search is incorporated for sparse recovery by a novel method, whose tractability follows from the properly defined cost models and pruning techniques. The proposed method is evaluated by both theoretical and empirical analyses, which clearly emphasize the improvements in the recovery accuracy. Next, we introduce an iterative two stage thresholding algorithm, where the forward step adds a larger number of nonzero elements to the sparse representation than the backward one removes. The presented simulation results reveal not only the recovery abilities of the proposed method, but also illustrate optimal choices for the step sizes. Finally, we propose a new mixed integer linear programming formulation for sparse recovery. Due to the equivalency of this formulation to the original problem, the solution is guaranteed to be correct when it can be solved in reasonable time. The simulation results indicate that the solution can be found easily under some reasonable assumptions, especially for signals with constant amplitude nonzero elements.

[1]  J. Tropp,et al.  SIGNAL RECOVERY FROM PARTIAL INFORMATION VIA ORTHOGONAL MATCHING PURSUIT , 2005 .

[2]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[3]  Mike E. Davies,et al.  Gradient Pursuits , 2008, IEEE Transactions on Signal Processing.

[4]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[5]  Wotao Yin,et al.  Sparse Signal Reconstruction via Iterative Support Detection , 2009, SIAM J. Imaging Sci..

[6]  Dimitri Kanevsky,et al.  Unscented compressed sensing , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[7]  Minh N. Do,et al.  Tree-Based Orthogonal Matching Pursuit Algorithm for Signal Reconstruction , 2006, 2006 International Conference on Image Processing.

[8]  D. Donoho,et al.  Redundant Multiscale Transforms and Their Application for Morphological Component Separation , 2004 .

[9]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[10]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  T. Cai,et al.  Limiting laws of coherence of random matrices with applications to testing covariance structure and construction of compressed sensing matrices , 2011, 1102.2925.

[12]  Joel A. Tropp,et al.  Simultaneous sparse approximation via greedy pursuit , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[13]  Jian Wang,et al.  Exact reconstruction of sparse signals via generalized orthogonal matching pursuit , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[14]  Justin Ziniel,et al.  Fast bayesian matching pursuit , 2008, 2008 Information Theory and Applications Workshop.

[15]  R. Kipp Martin,et al.  Large scale linear and integer optimization - a unified approach , 1998 .

[16]  Tong Zhang,et al.  Adaptive Forward-Backward Greedy Algorithm for Learning Sparse Representations , 2011, IEEE Transactions on Information Theory.

[17]  Volkan Cevher,et al.  Fast hard thresholding with Nesterov"s gradient method , 2010, NIPS 2010.

[18]  RaphaelBertram,et al.  Correction to "A Formal Basis for the Heuristic Determination of Minimum Cost Paths" , 1972 .

[19]  Tong Zhang,et al.  Sparse Recovery With Orthogonal Matching Pursuit Under RIP , 2010, IEEE Transactions on Information Theory.

[20]  Lawrence Carin,et al.  Bayesian Compressive Sensing , 2008, IEEE Transactions on Signal Processing.

[21]  Christian Jutten,et al.  Error correction via smoothed L0-norm recovery , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[22]  Richard G. Baraniuk,et al.  Recovery of Jointly Sparse Signals from Few Random Projections , 2005, NIPS.

[23]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[24]  G. Karabulut,et al.  Integrating flexible tree searches to orthogonal matching pursuit algorithm , 2006 .

[25]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[26]  Mário A. T. Figueiredo Adaptive Sparseness for Supervised Learning , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  J. Tropp Norms of Random Submatrices and Sparse Approximation , 2008 .

[28]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.

[29]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[30]  Simon Foucart,et al.  Stability and Robustness of Weak Orthogonal Matching Pursuits , 2012 .

[31]  Gabriel Peyré Best basis compressed sensing , 2010, IEEE Trans. Signal Process..

[32]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[33]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[34]  Volkan Cevher,et al.  Sparse Signal Recovery Using Markov Random Fields , 2008, NIPS.

[35]  P. Frossard,et al.  Tree-Based Pursuit: Algorithm and Properties , 2006, IEEE Transactions on Signal Processing.

[36]  S. Foucart,et al.  Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 , 2009 .

[37]  E. Candès,et al.  Error correction via linear programming , 2005, FOCS 2005.

[38]  Yonina C. Eldar,et al.  Block sparsity and sampling over a union of subspaces , 2009, 2009 16th International Conference on Digital Signal Processing.

[39]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[40]  Richard G. Baraniuk,et al.  Wavelet-domain compressive signal reconstruction using a Hidden Markov Tree model , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[41]  Dmitry M. Malioutov,et al.  Boolean compressed sensing: LP relaxation for group testing , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Jian Wang,et al.  Improved Recovery Bounds of Orthogonal Matching Pursuit using Restricted Isometry Property , 2012, ArXiv.

[43]  Sundeep Rangan,et al.  Orthogonal Matching Pursuit From Noisy Random Measurements: A New Analysis , 2009, NIPS.

[44]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[45]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[46]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[47]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[48]  David L. Donoho,et al.  Counting the Faces of Randomly-Projected Hypercubes and Orthants, with Applications , 2008, Discret. Comput. Geom..

[49]  D. Donoho For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[50]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[51]  Rick Chartrand,et al.  Nonconvex Compressed Sensing and Error Correction , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[52]  Bhaskar D. Rao,et al.  An Empirical Bayesian Strategy for Solving the Simultaneous Sparse Approximation Problem , 2007, IEEE Transactions on Signal Processing.

[53]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[54]  Marco F. Duarte,et al.  Compressive sensing recovery of spike trains using a structured sparsity model , 2009 .

[55]  Michael B. Wakin,et al.  Analysis of Orthogonal Matching Pursuit Using the Restricted Isometry Property , 2009, IEEE Transactions on Information Theory.

[56]  Kannan Ramchandran,et al.  Denoising by Sparse Approximation: Error Bounds Based on Rate-Distortion Theory , 2006, EURASIP J. Adv. Signal Process..

[57]  R.G. Baraniuk,et al.  Distributed Compressed Sensing of Jointly Sparse Signals , 2005, Conference Record of the Thirty-Ninth Asilomar Conference onSignals, Systems and Computers, 2005..

[58]  Michael Elad,et al.  Submitted to Ieee Transactions on Image Processing Image Decomposition via the Combination of Sparse Representations and a Variational Approach , 2022 .

[59]  Richard Baraniuk,et al.  Recovery of Clustered Sparse Signals from Compressive Measurements , 2009 .

[60]  Rick Chartrand,et al.  Fast algorithms for nonconvex compressive sensing: MRI reconstruction from very few data , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[61]  Kannan Ramchandran,et al.  Analysis of denoising by sparse approximation with random frame asymptotics , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[62]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[63]  David Furcy,et al.  Incremental Heuristic Search in AI , 2004, AI Mag..

[64]  Hakan Erdogan,et al.  A comparison of termination criteria for A∗OMP , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[65]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[66]  Richard G. Baraniuk,et al.  An Information-Theoretic Approach to Distributed Compressed Sensing ∗ , 2005 .

[67]  Hakan Erdogan,et al.  Forward-backward search for compressed sensing signal recovery , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[68]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[69]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[70]  Volkan Cevher,et al.  Hard thresholding with norm constraints , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[71]  Hassan Mansour,et al.  Support driven reweighted ℓ1 minimization , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[72]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[73]  Christian Jutten,et al.  A Fast Approach for Overcomplete Sparse Decomposition Based on Smoothed $\ell ^{0}$ Norm , 2008, IEEE Transactions on Signal Processing.

[74]  Jian Wang,et al.  On the Recovery Limit of Sparse Signals Using Orthogonal Matching Pursuit , 2012, IEEE Transactions on Signal Processing.

[75]  Jian Wang,et al.  Generalized Orthogonal Matching Pursuit , 2011, IEEE Transactions on Signal Processing.

[76]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[77]  Frederick Jelinek,et al.  Statistical methods for speech recognition , 1997 .

[78]  Volkan Cevher,et al.  Model-based compressive sensing for signal ensembles , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[79]  Bhaskar D. Rao,et al.  Application of tree-based searches to matching pursuit , 2001, 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221).

[80]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[81]  S. Mendelson,et al.  Uniform Uncertainty Principle for Bernoulli and Subgaussian Ensembles , 2006, math/0608665.

[82]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[83]  E. Candès The restricted isometry property and its implications for compressed sensing , 2008 .

[84]  David P. Wipf,et al.  A New View of Automatic Relevance Determination , 2007, NIPS.

[85]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[86]  David P. Wipf,et al.  Iterative Reweighted 1 and 2 Methods for Finding Sparse Solutions , 2010, IEEE J. Sel. Top. Signal Process..

[87]  Jared Tanner,et al.  Improved Bounds on Restricted Isometry Constants for Gaussian Matrices , 2010, SIAM J. Matrix Anal. Appl..

[88]  Richard G. Baraniuk,et al.  Distributed Compressed Sensing Dror , 2005 .

[89]  M. Rudelson,et al.  Geometric approach to error-correcting codes and reconstruction of signals , 2005, math/0502299.

[90]  M. Rudelson,et al.  On sparse reconstruction from Fourier and Gaussian measurements , 2008 .

[91]  Hakan Erdogan,et al.  A* orthogonal matching pursuit: Best-first search for compressed sensing signal recovery , 2010, Digit. Signal Process..

[92]  Aggelos K. Katsaggelos,et al.  Bayesian Compressive Sensing Using Laplace Priors , 2010, IEEE Transactions on Image Processing.

[93]  Bhaskar D. Rao,et al.  Latent Variable Bayesian Models for Promoting Sparsity , 2011, IEEE Transactions on Information Theory.

[94]  Stephen J. Wright,et al.  Computational Methods for Sparse Solution of Linear Inverse Problems , 2010, Proceedings of the IEEE.

[95]  E. Candes,et al.  11-magic : Recovery of sparse signals via convex programming , 2005 .

[96]  Richard G. Baraniuk,et al.  Fast reconstruction of piecewise smooth signals from random projections , 2005 .

[97]  M. Rudelson,et al.  Sparse reconstruction by convex relaxation: Fourier and Gaussian measurements , 2006, 2006 40th Annual Conference on Information Sciences and Systems.

[98]  Olgica Milenkovic,et al.  Sublinear compressive sensing reconstruction via belief propagation decoding , 2009, 2009 IEEE International Symposium on Information Theory.

[99]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[100]  Arian Maleki,et al.  Optimally Tuned Iterative Reconstruction Algorithms for Compressed Sensing , 2009, IEEE Journal of Selected Topics in Signal Processing.

[101]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[102]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[103]  Heung-No Lee,et al.  On the derivation of RIP for random Gaussian matrices and binary sparse signals , 2011, ICTC 2011.

[104]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing Signal Reconstruction , 2008, IEEE Transactions on Information Theory.

[105]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[106]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[107]  Michael Elad,et al.  A Plurality of Sparse Representations Is Better Than the Sparsest One Alone , 2009, IEEE Transactions on Information Theory.

[108]  Deanna Needell,et al.  Signal Recovery From Incomplete and Inaccurate Measurements Via Regularized Orthogonal Matching Pursuit , 2007, IEEE Journal of Selected Topics in Signal Processing.

[109]  Gabriel Peyré,et al.  Best basis denoising with non-stationary wavelet packets , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[110]  Piotr Indyk,et al.  Combining geometry and combinatorics: A unified approach to sparse signal recovery , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[111]  Rina Dechter,et al.  Generalized best-first search strategies and the optimality of A* , 1985, JACM.

[112]  Mike E. Davies,et al.  Stagewise Weak Gradient Pursuits , 2009, IEEE Transactions on Signal Processing.

[113]  Hakan Erdogan,et al.  Compressed sensing signal recovery via A* Orthogonal Matching Pursuit , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[114]  Jared Tanner,et al.  Explorer Compressed Sensing : How Sharp Is the Restricted Isometry Property ? , 2011 .

[115]  Thomas Blumensath,et al.  Accelerated iterative hard thresholding , 2012, Signal Process..

[116]  Michael E. Tipping,et al.  Fast Marginal Likelihood Maximisation for Sparse Bayesian Models , 2003 .

[117]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[118]  Rahul Garg,et al.  Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property , 2009, ICML '09.

[119]  Babak Hassibi,et al.  On the Reconstruction of Block-Sparse Signals With an Optimal Number of Measurements , 2008, IEEE Transactions on Signal Processing.

[120]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[121]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[122]  Yonina C. Eldar,et al.  Structured Compressed Sensing: From Theory to Applications , 2011, IEEE Transactions on Signal Processing.

[123]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.