Low-Dimensional Models for Dimensionality Reduction and Signal Recovery: A Geometric Perspective

We compare and contrast from a geometric perspective a number of low-dimensional signal models that support stable information-preserving dimensionality reduction. We consider sparse and compressible signal models for deterministic and random signals, structured sparse and compressible signal models, point clouds, and manifold signal models. Each model has a particular geometrical structure that enables signal information to be stably preserved via a simple linear and nonadaptive projection to a much lower dimensional space; in each case the projection dimension is independent of the signal's ambient dimension at best or grows logarithmically with it at worst. As a bonus, we point out a common misconception related to probabilistic compressible signal models, namely, by showing that the oft-used generalized Gaussian and Laplacian models do not support stable linear dimensionality reduction.

[1]  Piotr Indyk,et al.  Nearest-neighbor-preserving embeddings , 2007, TALG.

[2]  Jong Chul Ye,et al.  k‐t FOCUSS: A general compressed sensing framework for high resolution dynamic MRI , 2009, Magnetic resonance in medicine.

[3]  Richard G. Baraniuk,et al.  Distributed Compressed Sensing Dror , 2005 .

[4]  Michael B. Wakin,et al.  A multiscale framework for Compressive Sensing of video , 2009, 2009 Picture Coding Symposium.

[5]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[6]  Wei Lu,et al.  Real-time dynamic MR image reconstruction using Kalman Filtered Compressed Sensing , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[7]  Yue M. Lu,et al.  Sampling Signals from a Union of Subspaces , 2008, IEEE Signal Processing Magazine.

[8]  Lawrence Carin,et al.  Bayesian Compressive Sensing , 2008, IEEE Transactions on Signal Processing.

[9]  Mike E. Davies,et al.  Sampling Theorems for Signals From the Union of Finite-Dimensional Linear Subspaces , 2009, IEEE Transactions on Information Theory.

[10]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Union of Subspaces , 2008, ArXiv.

[11]  W. Marsden I and J , 2012 .

[12]  Babak Hassibi,et al.  On the Reconstruction of Block-Sparse Signals With an Optimal Number of Measurements , 2008, IEEE Transactions on Signal Processing.

[13]  Michael B. Wakin,et al.  A manifold lifting algorithm for multi-view compressive imaging , 2009, 2009 Picture Coding Symposium.

[14]  Kenneth L. Clarkson,et al.  Tighter bounds for random projections of manifolds , 2008, SCG '08.

[15]  R.G. Baraniuk,et al.  Compressive Sensing [Lecture Notes] , 2007, IEEE Signal Processing Magazine.

[16]  Allen Y. Yang,et al.  Estimation of Subspace Arrangements with Applications in Modeling and Segmenting Mixed Data , 2008, SIAM Rev..

[17]  Babak Hassibi,et al.  Recovering Sparse Signals Using Sparse Measurement Matrices in Compressed DNA Microarrays , 2008, IEEE Journal of Selected Topics in Signal Processing.

[18]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[19]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[20]  Sylvia Nasar 11. The Imbedding Problem for Riemannian Manifolds , 2002 .

[21]  Anupam Gupta,et al.  An elementary proof of the Johnson-Lindenstrauss Lemma , 1999 .

[22]  I. Daubechies,et al.  Tree Approximation and Optimal Encoding , 2001 .

[23]  H. Rauhut,et al.  Atoms of All Channels, Unite! Average Case Analysis of Multi-Channel Sparse Recovery Using Greedy Algorithms , 2008 .

[24]  Chinmay Hegde,et al.  Random Projections for Manifold Learning , 2007, NIPS.

[25]  Richard G. Baraniuk,et al.  Compressive Sensing , 2008, Computer Vision, A Reference Guide.

[26]  Geoffrey E. Hinton,et al.  Modeling the manifolds of images of handwritten digits , 1997, IEEE Trans. Neural Networks.

[27]  B. Dundas,et al.  DIFFERENTIAL TOPOLOGY , 2002 .

[28]  Horst Alzer,et al.  On some inequalities for the incomplete gamma function , 1997, Math. Comput..

[29]  Sanjoy Dasgupta,et al.  Random projection trees and low dimensional manifolds , 2008, STOC.

[30]  Herbert A. David,et al.  Order Statistics , 2011, International Encyclopedia of Statistical Science.

[31]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[32]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[33]  David B. Dunson,et al.  Multitask Compressive Sensing , 2009, IEEE Transactions on Signal Processing.

[34]  J. Tropp Algorithms for simultaneous sparse approximation. Part II: Convex relaxation , 2006, Signal Process..

[35]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[36]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[37]  Lawrence Carin,et al.  Exploiting Structure in Wavelet-Based Bayesian Compressive Sensing , 2009, IEEE Transactions on Signal Processing.

[38]  Justin Ziniel,et al.  Fast bayesian matching pursuit , 2008, 2008 Information Theory and Applications Workshop.

[39]  Massimo Fornasier,et al.  Recovery Algorithms for Vector-Valued Data with Joint Sparsity Constraints , 2008, SIAM J. Numer. Anal..

[40]  Volkan Cevher,et al.  Learning with Compressible Priors , 2009, NIPS.

[41]  Robert D. Nowak,et al.  Wavelet-based statistical signal processing using hidden Markov models , 1998, IEEE Trans. Signal Process..

[42]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[43]  Dimitris Achlioptas,et al.  Database-friendly random projections , 2001, PODS.

[44]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.

[45]  Anil K. Jain,et al.  Markov random fields : theory and application , 1993 .

[46]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[47]  Chinmay Hegde,et al.  A Theoretical Analysis of Joint Manifolds , 2009, ArXiv.

[48]  R. DeVore,et al.  Instance-optimality in probability with an ℓ1-minimization decoder , 2009 .

[50]  Richard Baraniuk,et al.  Recovery of Clustered Sparse Signals from Compressive Measurements , 2009 .

[51]  Joel A. Tropp,et al.  ALGORITHMS FOR SIMULTANEOUS SPARSE APPROXIMATION , 2006 .

[52]  Martin J. Wainwright,et al.  Image denoising using scale mixtures of Gaussians in the wavelet domain , 2003, IEEE Trans. Image Process..

[53]  J. Nash The imbedding problem for Riemannian manifolds , 1956 .

[54]  Minh N. Do,et al.  A Theory for Sampling Signals from a Union of Subspaces , 2022 .

[55]  Michael E. Tipping Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[56]  Baoxin Li,et al.  Compressive imaging of color images , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[57]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[58]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[59]  David B. Dunson,et al.  Multi-Task Compressive Sensing , 2007 .

[60]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[61]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[62]  E. Candès The restricted isometry property and its implications for compressed sensing , 2008 .

[63]  R. Stephenson A and V , 1962, The British journal of ophthalmology.

[64]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[65]  P. Bühlmann,et al.  The group lasso for logistic regression , 2008 .

[66]  Marco F. Duarte,et al.  Compressive sensing recovery of spike trains using a structured sparsity model , 2009 .

[67]  Rebecca Willett,et al.  Compressive coded aperture video reconstruction , 2008, 2008 16th European Signal Processing Conference.

[68]  Babak Hassibi,et al.  On the reconstruction of block-sparse signals with an optimal number of measurements , 2009, IEEE Trans. Signal Process..

[69]  Georgios B. Giannakis,et al.  RLS-weighted Lasso for adaptive estimation of sparse signals , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[70]  Jun Zhang,et al.  On Recovery of Sparse Signals Via $\ell _{1}$ Minimization , 2008, IEEE Transactions on Information Theory.

[71]  Justin K. Romberg,et al.  Bayesian tree-structured image modeling using wavelet-domain hidden Markov models , 2001, IEEE Trans. Image Process..

[72]  Richard G. Baraniuk,et al.  The smashed filter for compressive classification and target recognition , 2007, Electronic Imaging.

[73]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[74]  Pankaj K. Agarwal,et al.  Embeddings of surfaces, curves, and moving points in euclidean space , 2007, SCG '07.

[75]  Rayan Saab,et al.  Stable sparse approximations via nonconvex optimization , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[76]  Jerome M. Shapiro,et al.  Embedded image coding using zerotrees of wavelet coefficients , 1993, IEEE Trans. Signal Process..

[77]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[78]  J. Tropp,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, Commun. ACM.

[79]  Andrej Yu. Garnaev,et al.  On widths of the Euclidean Ball , 1984 .

[80]  Volkan Cevher,et al.  Sparse Signal Recovery Using Markov Random Fields , 2008, NIPS.

[81]  Richard G. Baraniuk,et al.  Random Projections of Smooth Manifolds , 2009, Found. Comput. Math..

[82]  Richard G. Baraniuk,et al.  The multiscale structure of non-differentiable image manifolds , 2005, SPIE Optics + Photonics.

[83]  M. Wakin Manifold-Based Signal Recovery and Parameter Estimation from Compressive Measurements , 2010, 1002.1247.