On the Global-Local Dichotomy in Sparsity Modeling

The traditional sparse modeling approach, when applied to inverse problems with large data such as images, essentially assumes a sparse model for small overlapping data patches and processes these patches as if they were independent from each other. While producing state-of-the-art results, this methodology is suboptimal, as it does not attempt to model the entire global signal in any meaningful way—a nontrivial task by itself.

[1]  Michael Elad,et al.  Working Locally Thinking Globally - Part II: Stability and Algorithms for Convolutional Sparse Coding , 2016, ArXiv.

[2]  Gordon Wetzstein,et al.  Fast and flexible convolutional sparse coding , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[4]  Mike E. Davies,et al.  Sampling Theorems for Signals From the Union of Finite-Dimensional Linear Subspaces , 2009, IEEE Transactions on Information Theory.

[5]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[6]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Michael Elad,et al.  Working Locally Thinking Globally - Part I: Theoretical Guarantees for Convolutional Sparse Coding , 2016, ArXiv.

[8]  Emmanuel J. Candès,et al.  Modern statistical estimation via oracle inequalities , 2006, Acta Numerica.

[9]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[10]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[11]  Joel A. Tropp,et al.  ALGORITHMS FOR SIMULTANEOUS SPARSE APPROXIMATION , 2006 .

[12]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[13]  Yonina C. Eldar,et al.  Block sparsity and sampling over a union of subspaces , 2009, 2009 16th International Conference on Digital Signal Processing.

[14]  Michael Elad,et al.  Patch-disagreement as away to improve K-SVD denoising , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Lei Zhang,et al.  Convolutional Sparse Coding for Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[17]  Gitta Kutyniok,et al.  Sparse Recovery From Combined Fusion Frame Measurements , 2009, IEEE Transactions on Information Theory.

[18]  R. Quiroga.  Spike sorting , 2012, Current Biology.

[19]  Stéphane Mallat,et al.  Solving Inverse Problems With Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity , 2010, IEEE Transactions on Image Processing.

[20]  Mike E. Davies,et al.  Sparse and shift-Invariant representations of music , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[21]  Minh N. Do,et al.  A Theory for Sampling Signals from a Union of Subspaces , 2022 .

[22]  Eero P. Simoncelli,et al.  Journal of Neuroscience Methods , 2022 .

[23]  Michael S. Lewicki,et al.  Efficient Coding of Time-Relative Structure Using Spikes , 2005, Neural Computation.

[24]  Michael Elad,et al.  Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding , 2017, IEEE Transactions on Signal Processing.

[25]  D. S. Arnon,et al.  Algorithms in real algebraic geometry , 1988 .

[26]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[28]  Volkan Cevher,et al.  Structured Sparsity: Discrete and Convex approaches , 2015, ArXiv.

[29]  Michael Elad,et al.  Multi-Scale Patch-Based Image Restoration , 2016, IEEE Transactions on Image Processing.

[30]  Sotirios A. Tsaftaris,et al.  Explicit Shift-Invariant Dictionary Learning , 2014, IEEE Signal Processing Letters.

[31]  Junzhou Huang,et al.  The Benefit of Group Sparsity , 2009 .

[32]  Joel A. Tropp,et al.  Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit , 2006, Signal Process..

[33]  Ajay N. Jain,et al.  Assembly of microarrays for genome-wide measurement of DNA copy number , 2001, Nature Genetics.

[34]  Yichuan Zhang,et al.  Advances in Neural Information Processing Systems 25 , 2012 .

[35]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[36]  Michael Elad,et al.  Expected Patch Log Likelihood with a Sparse Prior , 2014, EMMCVPR.

[37]  Max A. Little,et al.  Generalized methods and solvers for noise removal from piecewise constant signals. II. New methods , 2010, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[38]  Y-Lan Boureau,et al.  Learning Convolutional Feature Hierarchies for Visual Recognition , 2010, NIPS.

[39]  Michael Elad,et al.  Single Image Interpolation Via Adaptive Nonlocal Sparsity-Based Modeling , 2014, IEEE Transactions on Image Processing.

[40]  Jean Ponce,et al.  Sparse Modeling for Image and Vision Processing , 2014, Found. Trends Comput. Graph. Vis..

[41]  Lei Zhang,et al.  Nonlocally Centralized Sparse Representation for Image Restoration , 2013, IEEE Transactions on Image Processing.

[42]  Michael Elad,et al.  Convolutional Neural Networks Analyzed via Convolutional Sparse Coding , 2016, J. Mach. Learn. Res..

[43]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[44]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[45]  Michael Elad,et al.  Image denoising through multi-scale learnt dictionaries , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[46]  Michael Elad,et al.  Boosting of Image Denoising Algorithms , 2015, SIAM J. Imaging Sci..

[47]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[48]  Michael Elad,et al.  Sparse and Redundant Modeling of Image Content Using an Image-Signature-Dictionary , 2008, SIAM J. Imaging Sci..

[49]  Trung Le,et al.  Approximation Vector Machines for Large-scale Online Learning , 2016, J. Mach. Learn. Res..

[50]  Junzhou Huang,et al.  Learning with structured sparsity , 2009, ICML '09.

[51]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[52]  Jayaraman J. Thiagarajan,et al.  Shift-invariant sparse representation of images using learned dictionaries , 2008, 2008 IEEE Workshop on Machine Learning for Signal Processing.

[53]  Holger Rauhut,et al.  Uniform recovery of fusion frame structured sparse signals , 2014, ArXiv.

[54]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[56]  Michael Elad,et al.  Learning Multiscale Sparse Representations for Image and Video Restoration , 2007, Multiscale Model. Simul..

[57]  Anders P. Eriksson,et al.  Fast Convolutional Sparse Coding , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Peter G. Casazza,et al.  Finite Frames: Theory and Applications , 2012 .

[59]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[60]  Yonina C. Eldar,et al.  Robust Recovery of Signals From a Structured Union of Subspaces , 2008, IEEE Transactions on Information Theory.